mercredi 28 novembre 2018

Python and Beautiful soup || Regex with Varible before writting to file

I would love some assistance or help around an issue i'm currently having. I'm working on a little python scanner as a project. The libraries im current importing are:

requests
BeautifulSoup
re
tld

The exact issue is regarding 'scope' of the scanner. I'd like to pass a URL to the code and have the scanner grab all the anchor tags from the page, but only the ones relevant to the base URL, ignoring out of scope links and also subdomains.

Here is my current code, i'm by no means a programmer, so please excuse sloppy inefficient code.

import requests
from bs4 import BeautifulSoup
import re
from tld import get_tld, get_fld

#This Grabs the URL
print("Please type in a URL:")
URL = input()

#This strips out everthing leaving only the TLD (Future scope function)
def strip_domain(URL):
    global domain_name
    domain_name = get_fld(URL)
strip_domain(URL)


#This makes the request, and cleans up the source code
def connection(URL):
        r = requests.get(URL)
        status = r.status_code
        sourcecode = r.text
        soup = BeautifulSoup(sourcecode,features="html.parser")
        cleanupcode = soup.prettify()

        #This Strips the Anchor tags and adds them to the links array
        links = []
        for link in soup.findAll('a', attrs={'href': re.compile("^http://")}):
              links.append(link.get('href'))

        #This writes our clean anchor tags to a file
        with open('source.txt', 'w') as f:            
                for item in links:
                    f.write("%s\n" % item)

connection(URL)

The exact code issue is around the "for link in soup.find" section. I have been trying to parse the array for anchor tags the only contain the base domain, which is the global var "domain_name" so that it only writes the relevant links to the source txt file.

google.com accepted
google.com/file accepted
maps.google.com not written

If someone could assist me or point me in the right direction i'd appreciate it. I was also thinking it would be possible to write every link to the source.txt file and then alter it after removing the 'out of scope' links, but really thought it more beneficial to do it without having to create additional code.

Additionally, i'm not the strongest with regex, but here is someone that my help. This is some regex code to catch all variations of http, www, https

(^http:\/\/+|www.|https:\/\/)

To this I was going to append

.*{}'.format(domain_name)




Aucun commentaire:

Enregistrer un commentaire