Rusty Razor Blade

How curl_exec and urlencode killed my single sign on

April 16, 2008 – 11:02 am

If you do any work with single sign on, you’ll be familiar with the concept of exchanging tokens and validating against the authentication server using that token. One of the issues I’ve just run into which resulted in a huge headache is with urlencoding the result of a curl_exec that had a line ending. It’ easy to miss when it’s a longer string and you aren’t paying very close attention. This is a very simple example, and it still takes a second to realize there’s an extra character at the end.

php> echo urlencode(”test@str!ngw!th0u7\n”);
test%40str%21ngw%21th0u7%0A
php> echo urlencode(”test@str!ngw!th0u7″);
test%40str%21ngw%21th0u7

Basically, I’d recommend calling a trim() on any results you get back, unless you love newlines at the end for some reason.

Posted in curl, php | No Comments »

Awesome Awk Tutorial

April 15, 2008 – 12:29 am

This is exactly what I look for when I’m trying to find a tutorial. Thank you, Andrew M. Ross.

My favorite awk tutorial ever.

Posted in awk | No Comments »

Decimal vs Float in MySQL

April 1, 2008 – 5:26 pm

While I’ve known how floating points are stored, I didn’t know if decimal was stored any differently, or if it was more or less accurate. According to a post on the MySQL list:

Bruno Rodrigues Silva wrote:
> Dear all.
>
> The MySQL Manual inform that Decimal Data Type is used
> for represent exact-number, but like Float Point Data
> Type, Decimal use rounding case the fractional part is
> not sufficient. Therefore, what the difference?

Hi Bruno,

FLOAT rounds as floating point and DECIMAL rounds as you would expect it
to. Floating point arithmetic is not exact and that’s why they’ve called
DECIMAL an “exact packed decimal number” because apparently you can rely
on it’s roundings.

The primary use for DECIMAL is money, where floating point would fail
miserably and produce lots of errors (money leak) at the end of the month.

As floating point arithmetic has it’s own separated section on your
processor (unless you have a 386 or older computer) it would be *much*
faster than DECIMAL, so use it only if you are absolutely sure you need it.

cheers,
–renato

Not exactly breaking news, but I had never looked at decimal very closely. Good to know.

Posted in mysql | 1 Comment »

Problems Installing Subversion From Source On CentOS

March 26, 2008 – 12:23 am

I was setting up a new server for someone, and encountered this error while I was trying to build svn

/usr/bin/ld: cannot find -lexpat

Now, while I can do some things on that a sys admin can, I am by no means a sys admin. I have only installed svn a handful of times, and I didn’t know what this was.

First I installed expat from source. It didn’t help.

I then installed apr and apr-util. Didn’t help. I probably needed to follow these instructions.. but I didn’t.

Libraries have been installed in:
/usr/local/apr/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR’
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH’ environment variable
during execution
- add LIBDIR to the `LD_RUN_PATH’ environment variable
during linking
- use the `-Wl,–rpath -Wl,LIBDIR’ linker flag
- have your system administrator add LIBDIR to `/etc/ld.so.conf’

Decided I don’t care anymore about trying to figure out why it wasn’t working and just got the subversion dependencies from their site, untarred them into the svn folder as recommended. Everything compiled without issue.

On the make install, I got this:

/usr/local/src/subversion-1.4.6/subversion/svnversion/.libs/lt-svnversion: error while loading shared libraries: /usr/local/src/subversion-1.4.6/subversion/libsvn_delta/.libs/libsvn_delta-1.so.0: cannot restore segment prot after reloc: Permission denied
make: *** [revision-install] Error 127

Googling that error got me to this site, where it suggested SELinux might be breaking it. I disabled it and rebooted the machine. The install proceeded without issue at this point.

Posted in linux, svn | 5 Comments »

Report space remaining on disk on unix login

March 23, 2008 – 1:04 pm

I get a kick out of stuff like this. Add this to your .bash_profile to be greeted with the remaining space on your disk.

df -h | awk ‘NR==2 {print “Space available ” $4}’

This will actually change depending on which version of df you’re using (i think). I get different output on my Mac than I do on a CentOS machine - df seems to auto wrap lines on long drive names, so when I check the space on the CentOS server I need to change the line number (NR goes to 3) and the column to print (down to 3).

On my mac, this is what I get:

haddad:~ jhaddad$ df -h
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/disk0s2   149Gi  144Gi  4.9Gi    97%    /
devfs          106Ki  106Ki    0Bi   100%    /dev
fdesc          1.0Ki  1.0Ki    0Bi   100%    /dev
map -hosts       0Bi    0Bi    0Bi   100%    /net
map auto_home    0Bi    0Bi    0Bi   100%    /home

haddad:~ jhaddad$ df -h | awk 'NR==2 {print "Space available " $4}'
Space available 4.9Gi

I should probably clear some stuff off my drive.

Posted in awk, tips, unix | No Comments »

BBEdit, I love you, but please can we have a few more things?

March 15, 2008 – 10:35 am

BBEdit is made by Bare Bones Software. For text editing, I’ve never used anything better. I swear by it. It’s got absolutely amazing search / replace, regex support, shell worksheets, and the fact that I can write a plugin in pretty much any language ever made is unbelievable.

That being said, it’s still missing a few things that I’d love to see. I know I sound greedy, but I did pay about 180 for a text editor.

1. I don’t understand the Disk Browser. Why would I want a read only view of my files? It looks exactly like an editor, why can’t I just edit the files inline? Please make the disk browser functional and let me edit my stuff.

2. Better code completion. This isn’t entirely accurate, since you can use snippets to sort of do the code completion. I’ll be honest, if it had TAB auto completion with a drop down (like Visual Studio), I’d be willing to write the needed scrips to auto create snippets libraries based on PHP projects. Throw it on a shortcut key and we’ll be in business.

3. Enhanced split window mode. How about the ability to have 2 different documents in 1 split window?

4. To elaborate on #3, I’d also like to be able to use a shell directly in bbedit, as the second half of a split window. Or a shell worksheet works. I’d be cool with that.

5. Adding to #1 - one thing I like about Eclipse is that it reads in my directory and lets me quick open files with apple-shift R, then I can just type in the filename and it regex matches. It would be nice if when I’m using my editable disk browser I could do this.

Till then, I guess it’s a mix of Eclipse and BBEdit…

Posted in bbedit | No Comments »

How to Install the Lucene Search Engine using Solr

March 15, 2008 – 10:01 am

I’m going to go through the steps necessary to install and start using Solr. I’ve always been interested in trying out Lucene, but I never felt like dealing with writing my own wrapper around the classes. Solr simplifies this by creating a fully working search engine as a web service.

Let’s get started. You’ll need to check to see if Java is up to date. Run the below to find out.

java -version

You need to be running at least Java 1.5. Next is to check to see if Ant is installed. Do that with.

ant -version

I’m doing this on my Mac using Leopard. Here’s some instructions on setting up Ant if you aren’t using Leopard.

Java was up to date, and ant is built in. Sweet. However, I need to install JUnit. I’ll do that first.

Download JUnit. You’ll need to put it somewhere that’s accessible by the $CLASSPATH variable. There’s more information on the JUnit FAQ.

I threw it in /usr/share and left the name as junit-4.4.jar and set my CLASSPATH to point to that file (not the directory)

export CLASSPATH=$CLASSPATH:/usr/share/junit-4.4.jar

I ran that, as well as put it in my /etc/bashrc file (which you must be root to edit) so I don’t have to deal with it again.

Compile Solr

Switch back to the directory containing the solr files, and run:

ant compile

You should see something like this:

Buildfile: build.xml

init-forrest-entities:

checkJunitPresence:

compile:
[javac] Compiling 185 source files to /Users/jhaddad/src/apache-solr-1.2.0/build
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.

BUILD SUCCESSFUL
Total time: 3 seconds

I then ran:

ant dist

Which outputted something like this:

Buildfile: build.xml

init-forrest-entities:

checkJunitPresence:

compile:

make-manifest:
[mkdir] Created dir: /Users/jhaddad/src/apache-solr-1.2.0/build/META-INF

dist-jar:
[jar] Building jar: /Users/jhaddad/src/apache-solr-1.2.0/dist/apache-solr-1.2.1-dev.jar

dist-war:
[war] Building war: /Users/jhaddad/src/apache-solr-1.2.0/dist/apache-solr-1.2.1-dev.war

dist:

BUILD SUCCESSFUL
Total time: 0 seconds

You can run the example by going to the example directory and running

java -jar start.jar

Then go here: http://localhost:8983/solr/admin/ and check out your admin.

Load a few sample docs by going here:

/example/exampledocs

and running

java -jar post.jar solr.xml monitor.xml

I will post a follow up on how to get Solr running in Tomcat, as well as examples on how to use the server.

Posted in java, lucene, search, solr | 3 Comments »

Western Digital Drive with Leopard - “File system formatter failed.”

February 26, 2008 – 11:59 pm

I just bought a 1TB Western Digital drive. I am stoked.

However, I tried to format the drive on my Mac (Leopard) and got the error “File system formatter failed” when I tried to format the disk as MacOS Extended (Journaled). Not cool.

I found this forum thread which suggested using multiple partitions to solve the problem. When I was trying this, I went into options and changed the partition scheme from Master Boot Record to GUID partition table. It formatted fine.

To test things further, I changed the scheme back to 1 partition, and left the format as GUID partition table. This time it worked flawlessly.

The GUID type says it will not work as a start up disk MacOS older than 10.4. So if you’re still on Panther you can’t boot up off my backup drive. Try not to cry too much.

Posted in mac, tips | 9 Comments »

Executing multiple curl requests in parallel with PHP and curl_multi_exec

February 20, 2008 – 4:17 pm

Let’s get one thing out in the open. Curl is sweet. It does it’s job very well, and I’m absoutely thrilled it exists.

If you’re using curl in your PHP app to make web requests, you’ve probably realized that by doing them one after the other, the total time of your request is the sum of all the requests put together. That’s lame.

Unfortunately using the curl_multi_exec is poorly documented in the PHP manual.

Let’s say that your app is hitting APIs from these servers:

Google: .1s
Microsoft: .3s
rustyrazorblade.com: .5s

Your total time will be .9s, just for api calls.

By using curl_multi_exec, you can execute those requests in parallel, and you’ll only be limited by the slowest request, which is about .5 sec to rustyrazorblade in this case, assuming your download bandwidth is not slowing you down.

Sample code:

$nodes = array('http://www.google.com', 'http://www.microsoft.com', 'http://www.rustyrazorblade.com');
$node_count = count($nodes);

$curl_arr = array();
$master = curl_multi_init();

for($i = 0; $i < $node_count; $i++)
{
	$url =$nodes[$i];
	$curl_arr[$i] = curl_init($url);
	curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, true);
	curl_multi_add_handle($master, $curl_arr[$i]);
}

do {
    curl_multi_exec($master,$running);
} while($running > 0);

echo "results: ";
for($i = 0; $i < $node_count; $i++)
{
	$results = curl_multi_getcontent  ( $curl_arr[$i]  );
	echo( $i . "\n" . $results . "\n");
}
echo ‘done’;

It’s really not documented on php.net how to use curl_multi_getcontent, so hopefully this helps someone.

Posted in curl, php | 18 Comments »

External Libraries in XCode

February 6, 2008 – 5:44 pm

I need to compile something and use the MySQL C++ library. I have mysql and mysql++ already compiled, I won’t go over how to do that now.

I added the following code to the top of my source:


#include <mysql++.h>


I got an error

/Users/jhaddad/dev/search_engine/main.cpp:4:21: error: mysql++.h: No such file or directory

Not cool.

How to fix:

In XCode, open up the project settings (under the project menu). Go down to search paths, and you can change your Header search paths to the correct locations where you installed whatever you’re looking for. In this case, mine was /usr/local/includes and /usr/local/mysql/

Next time you try to recompile, you’ll get a different error, this time it should be during Linking. Might look something like the below.

“mysqlpp::Query::store(mysqlpp::SQLQueryParms&)”, referenced from:

Right click on your project in the left hand column (file listing), click “Add existing files”, then go to the prebuilt library (for me it was in /usr/local/lib), and add the file. You don’t have to copy it into the directory, you can just add it and it should work. Recompile and enjoy.

Edit: /usr/local/lib won’t be initially visible. Type command-shift-g and it’ll bring up a text field you can type a path into to go directly to a directory.

Posted in mac, mysql, tutorial | No Comments »