Visiting URL without file extension

I am writing a web crawler and am planning our how I will check if a URL exists on a domain. I was just wondering how I will check for a url on the website if I do not know the filetype. So usually if I were to visit a website I'd type in http://ift.tt/1jpx5k7 or .php etc. I was wondering if typing the file type is mandatory or if a web server will give me the resource regardless if I type the filetype or not. So would http://ift.tt/1lUBkSG always work?