Jump to content

* * * - - 1 votes

Ziff Davis Media pulls the Plug on FileFront?



91 replies to this topic

#81 Pille

Pille

    Member

  • Members
  • PipPipPip
  • 96 posts

Posted 26 March 2009 - 05:10 PM

Continue this childish arguing at a later time, right now we have to handle projects that require "co-operation". Tells you anything? Great.
Posted Image

#82 stewie

stewie

    Member

  • Members
  • Pip
  • 3 posts

Posted 26 March 2009 - 06:35 PM

http://arloria.net/i.../wiki/Main_Page <- Current list of files we have saved/are in the process of saving

#83 Apprentice

Apprentice

    Mighty pirate!

  • Members
  • 558 posts

Posted 27 March 2009 - 07:03 AM

Continue this childish arguing at a later time, right now we have to handle projects that require "co-operation". Tells you anything? Great.

Your collective "co-operation". Remember, I discarted Filesnetwork a long time ago . . .

#84 Pille

Pille

    Member

  • Members
  • PipPipPip
  • 96 posts

Posted 27 March 2009 - 09:42 AM

Who says I am talking about FileFront or this "Mod Network"? FileFront is a relict of the past in a few days, considering you were still able to run around bitching there on EliteForceFiles all the time I could imagine you do have an interest in that community outpost to keep existing though, in a new shape. So with all due respect, either you help like even Hocking is trying in his own way and look over your "FileFront evil, attack!!!" horizon which is completly irrelevant now, or you shut the hell up for once. Thank you :)
Posted Image

#85 John

John

    Computer Techie

  • RPG-X Tester
  • 285 posts

Posted 27 March 2009 - 02:59 PM

Important Question

Who is going to become the new version of EFfiles?

I have heard a number of people talking about it and I am unclear

We can get the EFreview site to act as a new site and we are very willing to but we don't want to comit resources if someone else is already doing a new site. So thats why I would liek to know.

---John

#86 Pille

Pille

    Member

  • Members
  • PipPipPip
  • 96 posts

Posted 27 March 2009 - 03:14 PM

It wouldn't be that wise to reveal everything in the public during the early planning phase. Contact me on MSN despite our past differencies, I will explain the whole thing to you. You still know my adress, right..
Posted Image

#87 The Tenth Doctor

The Tenth Doctor

    Admiral of the Fleet

  • Members
  • 597 posts

Posted 27 March 2009 - 03:25 PM

Pille is doing something, IKS Yo Mamma and us Filefront lot are doing something and I think a couple other "restorations" are in progress as well.

Luke Sutton, AKA 'The Doctor'

Media critic, writer, filmmaker, gamer, Trekker, and Time Lord. Site manager (still!) of Elite Force Files.com & creator of fan-film series Star Trek: UnitySatire is to me what winning elections isn't to Gordon Brown.

 

For links and info on my projects & investigations, view my blog:

 

http://lukesutton.blogspot.com/

 

Personal opinions are not endorsed by Filefront, Gateway Productions or Unity Starbase.

 


#88 Pille

Pille

    Member

  • Members
  • PipPipPip
  • 96 posts

Posted 27 March 2009 - 03:29 PM

And exactly that is not the case, not the way you put it at least. So please stop these speculations as they don't get anywhere.
Posted Image

#89 kwaz

kwaz

    Member

  • Members
  • Pip
  • 1 posts

Posted 28 March 2009 - 03:24 PM

If you want to backup for yourself here is a python script I found eariler and modified slightly.

You will need Python lots of bandwidth and basically a webserver to run it on. Just replace EDITME with the subdomain of the site you want to rip (there are two occurrences)

#!/usr/bin/env python
# A quick and VERY DIRTY filefront file ripper.
# LJ
import urllib
import os
import random



def getNames(sPage):
	""" Get a list of mod names from the page."""
	pPage = urllib.urlopen(sPage)
	
	# Find the link.
	print "Looking for links"
	lLines = pPage.readlines()
	lLinks = []
	for sLine in lLines:
			if "/file/" in sLine:
				lLinks.append(sLine)
	for sLink in lLinks:
		print sLink
	return lLinks
	
def doDownload(sPage, sFileName ="s"):
	""" Download a mod at a page with a gofetch link."""
	pPage = urllib.urlopen(sPage)
	
	# Find the link.
	print "Searching for Link"
	sDLine = None
	for sLine in pPage.readlines():
		if "gofetch" in sLine:
			sDLine = sLine.split("/file/gofetch/")[1].split("\"")[0]
			break
		elif "<big><b>Click here to Download the File</b></big></a>  </div><br>  <table summary=\"" in sLine:
			sName = sLine.split("<table summary=\"")[1].split("\"")[0]
			print sName
	if sDLine == "":
		return False
	
	# Download the file.
	print "Downloading Mod File."
	sModFile = "http://EDITME.filefront.com/file/gofetch/" + sDLine
	
	sCommand = "wget --user-agent='Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.5) Gecko/2007071317 Firefox/2.0.0.5' --load-cookies=cookies.txt"
	os.popen((sCommand + " --output-document='FULLFOLDERLOCATION"+sName+"_page.html' '" + sPage + "'") )
	os.popen((sCommand + " --output-document='FULLFOLDERLOCATION"+sName+"' '" + sModFile + "'") )
	
	print "Done"
	return sModFile


def downloadCatagoryPage(sCatPath):
	""" Download a catagory page.   NOTE: ensure you set the number of things per-page to 100 and do each page by hand """
	lLinks = getNames(sCatPath)
	for sLink in lLinks:
		try:
			doDownload(sLink)
		except:
			"error downloading" + str(sLink)


## HERE WE DOWNLOAD THE PAGES WE WANT

# The rest of fed ships main
downloadCatagoryPage("http://EDITME.filefront.com/sitemap_raw.txt")


If someone could get that rewritten to read a local copy of the sitemap_raw it would enable users to download a limited amount of files. Or have 2 people reorder the list and one starts from the A links and another stars from Z. Just a thought.

#90 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 16 November 2011 - 01:53 PM

Ah.... ah..... AH... CHOOO!

 
#!/usr/bin/env python
# A quick and VERY DIRTY filefront file ripper.
# LJ
#
# Enhanced by TiM of UberGames, November 2011
# (My first time with Python. Apologies in advance for the terrifying code. <img src='http://forums.ubergames.net/public/style_emoticons/<#EMO_DIR#>/happy.png' class='bbc_emoticon' alt='^_^' /><img src='http://forums.ubergames.net/public/style_emoticons/<#EMO_DIR#>/wink.png' class='bbc_emoticon' alt=';)' />
#
# Changes Made:
#   	- Rewrote parse code for FileFront frontend HTML templates (circa November 2011)
#   	- Upgraded to handle FileFront's download manager (circa November 2011)
#   	- Can download categories that span multiple pages
#   	- Has the option of automatically downloading any subcategories in the current category as well
#   	- Organizes downloaded files in a folder hierarchy matching the web site
#   	- Downloads any available screenshots attached to files
#
 
import urllib2
import os
import os.path
import random
import re
import socket
import time
 
# The local directory to download all of the files to (must be absolute)
kDownloadDirectory = "/Users/TiM/Desktop/FileFrontDownloads"
 
# The base URL for the target FileFront site
kSiteRoot = "http://eliteforce2.filefront.com"
 
# An array of all of the categories/subcategories pages that are to be downloaded
kCategoryLinks = [ '/files/Elite_Force;4index', '/files/Elite_Force_2;26index' ]
 
# An array of extra file links to download
kFileLinks = []
 
# If the category has sub-categories, it'll drop down and download those too
kDownloadSubCategories = True
 
# If the a file matching the nam of the file online is found, assume it's already downloaded and skip
kSkipExistingFiles = True
 
# For some reason, urllib2 will hang if the request times out
# This function will catch that and try again before terminating
def forceUrlOpen(pRequest, depth = 0):
        try:
                pPage = urllib2.urlopen( pRequest )
        except:
                if depth < 3:
                        print "Request timed out. Retrying."
                        return forceUrlOpen( pRequest, depth+1 )
                else:
                        print "Request Failed"
                        return null
 
        return pPage
        
 
def formatFileName(sFileName):
        if len(sFileName) <= 0:
                return ''
        
        # Replace all spaces with underscores
        sFileName = sFileName.replace( ' ', '_' )
 
        # Remove all other non-standard characters
        sFileName = re.sub( '[^a-zA-Z0-9_\-\.]', '', sFileName )
 
        return sFileName
 
def makeFolderPath( aFolderNames ):
        global kDownloadDirectory
 
        path = kDownloadDirectory
 
        # Loop through each folder, create it if it doesn't exist, and append to the final path
        for folderName in aFolderNames:
                folderName = formatFileName( folderName )
                path = path + '/' + '_' + folderName
 
                if os.path.exists( path ) == False:
             			os.popen( 'mkdir ' + path )
 
        return path
 
def getNames(sPage):
        # Get a list of mod names from the page.
        pPage = forceUrlOpen(sPage)
        pPageContent = str( pPage.read() )
      
        # Find the link.
        print "Looking for links..."
 
        lLinks = []
        
        # Looks like all of the file links are now rendered
        # without line-breaks. Better break out the regex!
        p = re.compile('<b><a class="size11" href="(/file/[^"]+)')
        reLinks = p.findall( pPageContent ) # searching for instance of '<a href="(/file/Lt_Kulhane;26594)">'
 
        # TiM: There are some redundant links like '#comments' that we can skip
        for sLink in reLinks:
                if sLink.find('#') == -1:
                        lLinks.append( sLink )
                        # print (sLink)
 
        # print len(lLinks)
 
        return lLinks
        
def doDownload(sPage, sDownloadRoute ):
        global kSiteRoot
        global kDownloadDirectory
 
        sCommand = "wget -c --user-agent='Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.5) Gecko/2007071317 Firefox/2.0.0.5' "
        # --load-cookies="+kDownloadDirectory+"/cookies.txt
 
        print ""
        print "***********************************************************"
        print "Downloading "+sPage
 
        # Download the file profile page
        sLink = ( kSiteRoot + sPage )
        pPageContent = forceUrlOpen(sLink).read()
        
        # Since it's possible for files to have both identical names and file names,
        # we'll load them down via their ID numbers
        fileID = sPage.split(';')[1]
 
        # Get the file name
        m = re.search( r'<span class="size14"><u>([^<]+)</u></span>', pPageContent )
        if m == False:
                return
        
        sFileName = m.group( 1 ).strip()
        
        m = re.search( r'Filename: <i>([^<]+)</i>', pPageContent )
        if m == False:
                return
                
        sFileStructureName = m.group( 1 ).strip()
 
        # All source files will be placed in a folder like 'Filename - ID'
        sFolderName = formatFileName( sFileName + '-' + str( fileID ) )
        
        sDownloadRoute = sDownloadRoute + '/' + sFolderName
        
        # If we've decided to skip existing files (eg resuming), check now
        if kSkipExistingFiles and os.path.exists( sDownloadRoute + '/' + sFileStructureName ):
                print "File already downloaded. Skipping."
                return
 
        os.popen( 'mkdir ' + sDownloadRoute )
 
        # Perform a search for any screenshots on the page and download any that have been found
        print ""
        print "Checking for Screenshots"
 
        p = re.compile( '<a href="/screenshots/File/[^"]+"[^>]+><img src="([^_]+_[0-9]+)t.jpg"' )
        lScreenshots = p.findall( pPageContent )
        print str( len(lScreenshots) ) + " screenshots found"
 
        if len( lScreenshots ) > 0:
                i = 1
                for sScreenshot in lScreenshots:
                        print "Downloading screenshot #" + str( i )
                        sScreenshot = sScreenshot + '.jpg'
 
                        os.chdir(sDownloadRoute)
                        os.popen( (sCommand + " --output-document='screenshot_"+str(i)+".jpg' " + sScreenshot) )
                        i = i+1
                
                print "Screenshot Downloads Completed"
                print ""
 
        time.sleep( 1 )
 
        # Find the link.
        print "Searching for File Download Link"
        
        m = re.search( r'<a class="size16" href="(/file/gofetch/[^"]+)', pPageContent )
        if m == False:
                print "Unable to locate Download Link"
                return
 
        # Jump to the FileFront file download prompt and download it
        print "Loading GameFront Download Portal"
        sDownloadPortalURL = (kSiteRoot + m.group( 1 )).strip()
 
        sDownloadPortalPage= forceUrlOpen( sDownloadPortalURL )
        sDownloadPortalPageContent = sDownloadPortalPage.read()
 
        # Preserve the session ID that gets generated with this request
        # Otherwise, the next one will kick us back home
        setCookie = sDownloadPortalPage.headers.getheader("Set-Cookie")
        sessId = setCookie[setCookie.index("=")+1:setCookie.index(";")]
        headers = { "Cookie": "PHPSESSID="+sessId }
 
        time.sleep( 1 )
 
        # Go to the actual Download page
        print "Loading GameFront Download Page"
        m = re.search( r'(http://www.gamefront.com/files/service/thankyou\?id=[0-9]+)', sDownloadPortalPageContent )
        sDownloadMirrorURL = m.group( 1 ).strip()
        
        # If the request times out, force a retry
        sDownloadMirrorPageContent = forceUrlOpen( urllib2.Request(sDownloadMirrorURL, headers=headers) ).read()
 
        # Grab the download link from the 'click here' button
        m = re.search( r'<br />If it does not, <a href="([^"]+)">click here', sDownloadMirrorPageContent )
        if m == False:
                return
 
        sModFile = m.group( 1 ).strip()
        sFileName = sModFile[ sModFile.rfind('/')+1:sModFile.rfind('?') ]
 
        # Download the file.
        os.chdir(sDownloadRoute)
        
        print "Downloading HTML File - " + sFolderName+"_page.html"
        os.popen((sCommand + " --output-document='"+sFolderName+"_page.html' '" + sLink+'x' + "'") ) # Adding an x expands all comments... apparently
        
        print "Downloading File - " + sFileName
        os.popen((sCommand + " --output-document='"+sFileName+"' '" + sModFile + "'") )
 
        print "File Download Complete"
        print "***********************************************************"
 
 
def downloadCatagoryPage(sCatPath):
        global kSiteRoot
        global kDownloadSubCategories
 
        # Download the base page
        sFullPath = (kSiteRoot + sCatPath)
        pPageContent = forceUrlOpen( sFullPath ).read()
 
        # calculate the subcategory structure from the root
        p = re.compile( '<a class="browser_head"[^>]+>([^<]+)' )
        parentCategories = p.findall( pPageContent )
        sDownloadPath = makeFolderPath( parentCategories )
 
        # if the list of files spans multiple pages, be sure to loop through each one
        start = 0;
        i = 1
        while True:
                #build the sCatPath in full, including root URL and sort parameters
                sFullCatPath = (kSiteRoot + sCatPath+"?&sort=name&name_direction=asc&limit=100&start=" + str(start) )
 
                lLinks = getNames(sFullCatPath)
                if lLinks == False:
                        print "Unable to parse page: "+sCatPath
                        return
 
                print str( len(lLinks) ) + " links found on page " + str( i )
 
                for sLink in lLinks:
                        try:
                                doDownload(sLink, sDownloadPath)
                        except:
                                "error downloading" + str(sLink)
 
                # terminate the loop if the list has less than 100 entries
                if len(lLinks) < 100:
                        break
 
                start += 100
                i = i+1
 
        # If requested, download the Sub-categories as well
        if kDownloadSubCategories == True:
                p = re.compile('<a class="size16" href="([^"]+)" title="[^"]+"><img src="/skins/icons/expand.gif"')
                subCategoryLinks = p.findall( pPageContent )
 
                if len( subCategoryLinks ) > 0:
                        print ""
                        print str( len(subCategoryLinks) ) + " subcategories found."
 
                        for subCategoryLink in subCategoryLinks:
                                print "Downloading subcategory: " + subCategoryLink
                                downloadCatagoryPage( subCategoryLink )
 
 
## *********************************************************************************
## FILEFRONT RIPPER ENTRY POINT
 
print ""
print "***********************************************************"
print "| FileFront File Ripper"
print "| November 2011 "
print "| By LJ "
print "| Changes/Enhancements by TiM (http://www.ubergames.net/)"
print "***********************************************************"
print ""
 
# Set request timeout to 30 seconds (else urllib2.urlopen hangs on timeout)
socket.setdefaulttimeout(30)
 
os.chdir( kDownloadDirectory )
 
# Download all of the pages
for sLink in kCategoryLinks:
        print "Downloading Category: " + sLink
        downloadCatagoryPage(sLink)
 
for sLink in kFileLinks:
        print "Downloading File: " + sLink
        doDownload( sLink, kDownloadDirectory )

>_>

-TiM

#91 GSIO01

GSIO01

    UberGames Developer

  • UberGames Developer
  • 1,021 posts

Posted 17 November 2011 - 08:40 AM

Good work ...

#92 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 18 November 2011 - 01:25 AM

Haha thanks. XD

I probably should mention... that script has the power to suck down every single file on a FileFront game site with one call to execute. Please use it responsibly. ;)




Reply to this topic



  

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users