7.28.1 breaks php curl functions. It appears to break curl_getinfo as it returns the wrong information, header/result sizes are completely wrong (eg. curl_getinfo($handle, CURLINFO_HEADER_SIZE)). Also when returning the "content" (curl_multi_getcontent($handle)) which should include headers (when setting (CURLOPT_HEADER, 1) and by default(iirc?)), there are none, the content returned appears correct. This may affect multicurl only, did not test with regular curl. rolling back to 7.28.0-r1 fixes it. Tested with PHP-5.3.19 (newest 5.3) and PHP-5.3.18.
Can you give me 0) Youre emerge --info 1) the version of php you emerged and the USE flags 2) A snippet of php code that demonstrates the problems.
Portage 2.2.0_alpha142 (default/linux/amd64/10.0/desktop/kde, gcc-4.6.3, glibc-2.16.0, 3.6.6-ck1 x86_64) ================================================================= System uname: Linux-3.6.6-ck1-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_6000+-with-gentoo-2.2 Timestamp of tree: Mon, 26 Nov 2012 10:30:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 distcc 3.1 x86_64-pc-linux-gnu [enabled] ccache version 3.1.8 [disabled] app-shells/bash: 4.2_p39 dev-java/java-config: 2.1.12 dev-lang/python: 2.7.3-r2, 3.2.3-r1 dev-util/ccache: 3.1.8 dev-util/cmake: 2.8.10.1 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.5 sys-apps/sandbox: 2.6 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.9.6-r3, 1.11.6, 1.12.5 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.6.3 sys-devel/gcc-config: 1.8 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.6 (virtual/os-headers) sys-libs/glibc: 2.16.0 Repositories: gentoo multimedia piczu vmware hedgehog Installed sets: @system ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=athlon64 -O2 -pipe -msse3" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa /usr/share/polkit-1/actions /usr/share/themes/oxygen-gtk/gtk-2.0" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=athlon64 -O2 -pipe -msse3" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distcc distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="xxftp://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ xxhttp://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ http://mirror.mcs.anl.gov/pub/gentoo/ http://gentoo.mirrors.tds.net/gentoo http://mirror.datapipe.net/gentoo" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en en_US" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/tmp/portage" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage/layman/multimedia /usr/local/portage/layman/piczu /usr/local/portage/layman/vmware /usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dnow 3dnowext X a52 aac accessibility acl acpi alsa amd64 apache2 apm arts audiofile bash-completion berkdb branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cups curl curlwrappers cxx dbus declarative device-mapper dri dts dv dvd dvdr dvdread emboss encode exif fam ffmpeg firefox flac fortran ftp gd gdbm gif gmp gnutls gphoto2 gpm gtk iconv icu ieee1394 imagemagick imap imlib ipv6 java javascript joystick jpeg jpeg2k kde kdexdeltas kipi lcms libnotify libwww lm_sensors mad matroska mbox memlimit mhash mime mmap mmx mmxext mng modules mozilla mp3 mp4 mpeg mplayer msn mudflap multilib mysql mysqli ncurses nls nntp nptl nptlonly odbc offensive ogg openal opengl openmp pam pango pcntl pcre pdf perl phonon php plasma png policykit posix ppds pppd python qt3support qt4 quicktime readline recode samba sdl session sharedmem simplexml sndfile snmp soap sockets spell sse sse2 ssl startup-notification svg symlink sysvipc tcpd theora threads tiff tokenizer truetype udev udisks unicode upower usb v4l v4l2 vdpau vhosts vorbis wifi wxwidgets x264 xcb xcomposite xine xinetd xml xpm xscreensaver xv xvid zlib" ALSA_CARDS="hda_intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="canon ptp2 samsung" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_US" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" USE_PYTHON="2.7 3.2" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
[ebuild R ] dev-lang/php-5.3.19:5.3 USE="apache2 bcmath bzip2 cli crypt ctype curl curlwrappers exif fileinfo filter ftp gd gdbm gmp hash imap ipv6 json mhash mysql mysqli nls pcntl phar posix readline session simplexml soap sockets spell ssl sysvipc threads tokenizer truetype unicode xml xmlreader xmlwriter zip zlib -berkdb -calendar -cdb -cgi -cjk -debug -doc -embed -enchant -firebird -flatfile -fpm (-frontbase) -iconv -inifile -intl -iodbc -kerberos (-kolab) -ldap -ldap-sasl -libedit -mssql -mysqlnd -oci8-instant-client -odbc -pdo -pic -postgres -qdbm -recode (-selinux) -sharedmem -snmp -sqlite -sqlite2 (-sybase-ct) -tidy -wddx -xmlrpc -xpm -xsl" 0 kB as for the code, I found this debugging an ~40k multicurl class, no quick little snippits available. however this should only take about 10 lines of code to test.
(In reply to comment #3) > > as for the code, I found this debugging an ~40k multicurl class, no quick > little snippits available. however this should only take about 10 lines of > code to test. I'm not a php coder, so if you can produce those 10 lines, and tell me what to expect, it would save me time figuring it out.
I can't seem to recreate it now, maybe I'm crazy. I'll keep looking into it.
Finally, tracked down the exact problem. <?php $cm = curl_multi_init(); $stuffs = curl_init(); $curl_options = array( CURLOPT_URL => 'http://www.google.ca', CURLOPT_RETURNTRANSFER => 1, CURLOPT_SSL_VERIFYHOST => 1, // these are the problem CURLOPT_SSL_VERIFYPEER => 1, // toggle 0/1 to test CURLOPT_HEADER => 1 ); curl_setopt_array($stuffs, $curl_options); curl_multi_add_handle($cm, $stuffs); do { curl_multi_exec($cm, $running); } while($running > 0); $content = curl_multi_getcontent($stuffs); $info = curl_getinfo($stuffs); curl_multi_remove_handle($cm, $stuffs); curl_multi_close($cm); print_r($info); // cuts off too much, the header size is incorrect echo mb_substr($content, $info['header_size']); ?> Test with newest curl, then roll back a version and test. I guess it was a little more then 10 lines :)
Thanks for the opporutnity to teach myself some php :) But I can't seem to reproduce this. First, I change the test script a bit: <?php $cm = curl_multi_init(); $stuffs = curl_init(); $curl_options = array( CURLOPT_URL => 'https://ddl.dyc.edu', CURLOPT_RETURNTRANSFER => 1, CURLOPT_SSL_VERIFYHOST => 0, // these are the problem CURLOPT_SSL_VERIFYPEER => 0, // toggle 0/1 to test CURLOPT_HEADER => 1 ); curl_setopt_array($stuffs, $curl_options); curl_multi_add_handle($cm, $stuffs); do { curl_multi_exec($cm, $running); } while($running > 0); $content = curl_multi_getcontent($stuffs); $info = curl_getinfo($stuffs); curl_multi_remove_handle($cm, $stuffs); curl_multi_close($cm); //print_r($info); echo "\n"; echo "****\n"; echo "\n"; echo mb_substr($content, 0, $info['header_size']); ?> 1) I'm using a site which I run. 2) Your original mb_substr was incorrect. It had incorrect limits for just the header. Anyhow, using either curl-7.26.0 or curl-7.28.0-r1, with either https or http, or either CURLOPT_SSL_VERIFYHOST => 0, CURLOPT_SSL_VERIFYPEER => 0, or CURLOPT_SSL_VERIFYHOST => 1, CURLOPT_SSL_VERIFYPEER => 1, I always get the following: Array ( [url] => https://ddl.dyc.edu [content_type] => text/html [http_code] => 200 [header_size] => 223 [request_size] => 50 [filetime] => -1 [ssl_verify_result] => 0 [redirect_count] => 0 [total_time] => 0.516922 [namelookup_time] => 0.002 [connect_time] => 0.032995 [pretransfer_time] => 0.452932 [size_upload] => 0 [size_download] => 13720 [speed_download] => 26541 [speed_upload] => 0 [download_content_length] => 13720 [upload_content_length] => 0 [starttransfer_time] => 0.516922 [redirect_time] => 0 [certinfo] => Array ( ) [primary_ip] => 67.151.215.230 [primary_port] => 443 [local_ip] => 192.168.100.192 [local_port] => 41588 [redirect_url] => ) **** HTTP/1.1 200 OK Date: Sat, 01 Dec 2012 17:02:07 GMT Server: Apache Last-Modified: Wed, 31 Oct 2012 19:57:08 GMT ETag: "100f6c-3598-4cd604ec36ec0" Accept-Ranges: bytes Content-Length: 13720 Content-Type: text/html Can you try at your end and see if you're getting what I'm getting? Maybe its site specific, but even www.google.ca gave me the same results.
2) Your original mb_substr was incorrect. It had incorrect limits for just the header. Actually no, I was trying to remove the header not just cut out the header itself. And you may need to use something other then google.ca, which is local for me, google.com fowards me to .ca so I used that. Try with whatever google domain doesn't forward you. Use my code and look at what is returned, the older curl cuts out the headers properly and the newer one cuts into the html returned by google.
(In reply to comment #8) > 2) Your original mb_substr was incorrect. It had incorrect limits for just > the header. > > Actually no, I was trying to remove the header not just cut out the header > itself. Oh sorry. Even so, I still can't get a differnce. > > And you may need to use something other then google.ca, which is local for > me, google.com fowards me to .ca so I used that. Try with whatever google > domain doesn't forward you. > > Use my code and look at what is returned, the older curl cuts out the > headers properly and the newer one cuts into the html returned by google. I used exactly your script but changed google.ca to google.com, no difference. Can you post the difference that you're getting at your end.
script used - http://pastebin.com/HX5SKEXt 7.28.0 - 0s - http://pastebin.com/Bvbq5Y9Q 7.28.0 - 1s - http://pastebin.com/sYwstBnU 7.28.1 - 0s - http://pastebin.com/JVVbzwAy 7.28.1 - 1s - http://pastebin.com/arP00Y03
(In reply to comment #10) > script used - http://pastebin.com/HX5SKEXt > 7.28.0 - 0s - http://pastebin.com/Bvbq5Y9Q > 7.28.0 - 1s - http://pastebin.com/sYwstBnU > 7.28.1 - 0s - http://pastebin.com/JVVbzwAy > 7.28.1 - 1s - http://pastebin.com/arP00Y03 Those are the result. I'm not getting what I'm supposed to be seeing in those results that is the problem. So show me the diffs and point to "here" and what it should be.
7.28.1 - 1s - http://pastebin.com/arP00Y03 <-- wrong the others start with "<!doctype html><html", this one cuts into the beginning.
(In reply to comment #12) > 7.28.1 - 1s - http://pastebin.com/arP00Y03 <-- wrong > > the others start with "<!doctype html><html", this one cuts into the > beginning. My apologies for being such a blockhead! I kept missing that. I can confirm that its happening with every page and its actually a variable about.
(In reply to comment #12) > 7.28.1 - 1s - http://pastebin.com/arP00Y03 <-- wrong > > the others start with "<!doctype html><html", this one cuts into the > beginning. Okay CURLOPT_SSL_VERIFYHOST=1 is deprecated, but this is still a misbehavior. The upstream bug is at https://sourceforge.net/tracker/?func=detail&aid=3594897&group_id=976&atid=100976 And the commits which stop support for CURLOPT_SSL_VERIFYHOST=1 are at https://github.com/bagder/curl/commit/da82f59b697310229ccdf66104d5d65a44dfab98 https://github.com/bagder/curl/commit/a1be8e7f9be2feff103f314cd8ea8a50a560e79e I'm not going to revert those commits on gentoo because there's actually a research paper showing "in the wild" abuse. See Daniel Haxx's blog post about it http://daniel.haxx.se/blog/2012/10/25/libcurl-claimed-to-be-dangerous/
The irony of all this is that I want to use "1" in my code, I was using it correctly to get around broken/bad certs. Instead this should throw a warning on using booleans. So they broke working code to accommodate bad code... bunch of stupids...
The curl people are blaming php. They closed the curl bug: https://sourceforge.net/p/curl/bugs/1172/ I've pushed things php's way: https://bugs.php.net/bug.php?id=63795
As mentioned in previous, comments, this bug was introduced when the support of the value 1 for CURLOPT_SSL_VERIFYHOST was removed in 7.28.1. In your code sample, you're using curl_setopt_array. How this function work internally is that it will loop over all your options and set them one by one on your curl handle using the libcurl curl_easy_setopt function. If one of this set fail, the function will break the iteration and will return false. In your case, when you're using libcurl 7.28.1, curl_setopt_array will fail when it will try to set CURLOPT_SSL_VERIFYHOST to 1, and then will not set CURLOPT_HEADER. The result of your curl_exec function will then not include the headers. Then when you remove the begining of your string to remove your header, you're in fact removing the begining of your content since there is no header included in the original string. If you move your CURLOPT_SSL_VERIFYHOST to the end of your array this should fix your problem. Can you confirm ? I started a discussion to php internal to see how we want to deal with this bug, so that the changes made to the last libcurl versions will impact as less code as possible.
(In reply to comment #17) > > If you move your CURLOPT_SSL_VERIFYHOST to the end of your array this should > fix your problem. Can you confirm ? > Yep. Confirmed. The following order WORKS: $curl_options = array( CURLOPT_URL => 'http://www.google.ca', CURLOPT_RETURNTRANSFER => 1, CURLOPT_SSL_VERIFYPEER => 1, CURLOPT_HEADER => 1, CURLOPT_SSL_VERIFYHOST => 1 ); The following does NOT WORK: $curl_options = array( CURLOPT_URL => 'http://www.google.ca', CURLOPT_RETURNTRANSFER => 1, CURLOPT_SSL_VERIFYPEER => 1, CURLOPT_SSL_VERIFYHOST => 1, CURLOPT_HEADER => 1 );
This issue isn't the order of the array, the issue is having the ability to ignore bad certificates with "CURLOPT_SSL_VERIFYHOST => 1". ok I'll admit my code example doesn't *properly* demonstrate the issue, but rearranging the array is not a fix, hell it's not even a workaround.
(In reply to comment #19) > This issue isn't the order of the array, the issue is having the ability to > ignore bad certificates with "CURLOPT_SSL_VERIFYHOST => 1". > > ok I'll admit my code example doesn't *properly* demonstrate the issue, but > rearranging the array is not a fix, hell it's not even a workaround. I don't think Pierrick is suggesting it as a fix or workaround, just a diagnostic test to confirm his idea. Let's see what comes out of the php internal's discussion.
(In reply to comment #19) > This issue isn't the order of the array, the issue is having the ability to > ignore bad certificates with "CURLOPT_SSL_VERIFYHOST => 1". > > ok I'll admit my code example doesn't *properly* demonstrate the issue, but > rearranging the array is not a fix, hell it's not even a workaround. I was just asking this to confirm my theory :) I never proposed this as a fix or anything else. Unfortunately, the possibility to ignore bad certificates is not available anymore in libcurl and there is nothing I can do about it. I can only adapt some code in the php/curl extension to deal with this libcurl change as well as possible. The proposed solution was explained here : http://news.php.net/php.internals/64351 This will not fix your problem in case of bad certificates but will fix the most common cases.
Obviously this is going to happen, perhaps for the better, who knows. However I will say this, I find it hard to believe that the need to disable verifying bad certs is so uncommon that the functionality can be removed entirely.
php/curl was modified to manage the CURLOPT_SSL_VERIFYHOST change as proposed on php internal mailing list.
(In reply to comment #23) > php/curl was modified to manage the CURLOPT_SSL_VERIFYHOST change as > proposed on php internal mailing list. thank you Pierrick!
(In reply to comment #23) > php/curl was modified to manage the CURLOPT_SSL_VERIFYHOST change as > proposed on php internal mailing list. This issue (or a relative of it) is coming up in php applications, like moodle. See bug #450744. I'm confused though whether after the fix in php head, wether applications can continue using CURLOPT_SSL_VERIFYHOST = 1 or whether they have to switch to = 2?
(In reply to comment #25) > (In reply to comment #23) > > php/curl was modified to manage the CURLOPT_SSL_VERIFYHOST change as > > proposed on php internal mailing list. > > This issue (or a relative of it) is coming up in php applications, like > moodle. See bug #450744. I'm confused though whether after the fix in php > head, wether applications can continue using CURLOPT_SSL_VERIFYHOST = 1 or > whether they have to switch to = 2? cURL removed support for CURLOPT_SSL_VERIFYHOST as of 7.28.1, so if your php/curl is compiled with libcurl >= 7.28.1 you will need to use 2 instead of 1. If you're using the 1 value, php will use 2 and trigger a notice to notify the user that the 2 value is used because 1 is not supported anymore. Cf : http://git.php.net/?p=php-src.git;a=commitdiff;h=517f800277a11d6ce05b0e1afcd0e76dc544d452
