Of course, I'm not saying that you're wrong. If there were never any people to argue that something "isn't art," then we artists would get pretty lazy, no?
Here's something to help illustrate what I was trying to explain above:
http://www.cl.cam.ac.uk/~ig206/oak_tree.html
Click on the title
"An oak tree" in the first sentence to see this guy's "work," then read the Q & A. It's a good laugh, but it also makes you think; I take it that this was Mr. Craig-Martin's artistic intention all along.