Hmmm, my first handheld was the Merlin
http://en.wikipedia.org/wiki/Merlin_(game)
It has about 6 games on it and was really fun (at the time, being the 1980s).
Not sure if it was the first handheld console, but it must have been out there with the first.
Other than that, Well the Original Gameboy was a classic of it's time too. I played it to death.
Still, if I had to choose just one, at the end of the day, for the things that are important for a handheld as follows: battery life, portability, well protected, lots of good games, a decent backlight. For me, it has to be the DS Lite.
The Touchscreen really swings it for me though.
If the PSP had a touchscreen and say 10 games I really liked I would have said that instead.