JVDE X10 Forum

General Discussion => X10 General Topics => Topic started by: Tuicemen on August 05, 2013, 09:36:15 AM

Title: Resurecting x10's forum?
Post by: Tuicemen on August 05, 2013, 09:36:15 AM
Many users are interested in having this info saved.

What exactly should be salvaged?
Can this legally be done?
Would this be opening a can of worms?
So many questions ::)

If someone can get the forum rebuilt there are some that can access all parts of it (myself included)
The X10forum was run original using Simple Machines 1.1.2 if memory serves me right.
It also had some customization done that makes it difficult to upgrade.
It was however updated to 1.1.4  with a lot of work.

I have someone looking at the files that was familiar with the customization of the forum.
What will result from it is any ones guess.
Title: Re: Resurecting x10's forum?
Post by: Jeff on August 05, 2013, 10:02:32 AM
It is starting to look like I will not be able to host the X10 Forum archive after all.  It is apparently just too big for the level of service that I have.

There are about 120,000 files, and the unpacked size is almost 6GB.  Uploading individual files was terminated at 1GB, so I tried to unpack the archive directly on the server.  That process terminated at 760MB.

What I will do for now is host the packed archive, but even that is over 300MB.  I am hesitant to make it publicly available to keep the bandwidth demands reasonable.

I'm going to try to identify the files associated with just the automation and troubleshooting boards, and see if I can upload just that portion.

Jeff
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 05, 2013, 10:24:05 AM
I managed to unpack the files and can view all sections viewable to a guest.
However I'm unable to logon and edit or post this is most likely because of the customization done to the forum as it looks for info from another location which isn't accessible.
I've started to upload all files to my server so users may be able to at least look up old info.
That is if I don't run into the same issues as you Jeff.
In any case there are others working on this and my hopes are high that the old X10 Web master can access everything to make it a working forum again.

All isn't lost! ;)
 
Title: Re: Resurecting x10's forum?
Post by: Jeff on August 05, 2013, 11:22:33 AM
It doesn't appear feasible to pair down the size to just the automation boards.  While it is easy to identify each of the indexes (Airpad boards are 130-135), the individual topics themselves have random numbers - probably based on when that topic was created in the sequence.  So it is not possible to sort the topics into groups for each of the boards.  There may be some tool out there that can do that, but it is impossible to do manually.

I will try to resurrect my own "child board" on Powerline Control Reliability, but even that encompasses almost a hundred files.  The more popular boards have thousands of files.

Sorry this isn't working out as I had hoped.

Jeff
Title: Re: Resurecting x10's forum?
Post by: noam on August 05, 2013, 11:30:56 AM
Jeff -
Thanks for trying.
I know that Tuicemen (and maybe others) are working on a better way to do it. As individual HTML files, these forums can get rather large, especially if they were captured using a crawler that follows every link, even ones that were already covered another way.
If we can get a "real" export of the forums (in their native format), I'm guessing that would be a lot smaller, and could probably be imported into a new SMF instance more easily.

Since the old forums can be still be accessed by IP, there is hope that one of the old admins might be able to pull a full backup of the old forums.
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 05, 2013, 01:39:09 PM
The best way to get this is using FileZila in binary mode and download the forums complete folder and all sub folders.
Once that is done a new file needs to be added to repair info in the setup file that points to the new location of the folders.
Since I have the same files as Jeff it looks like I may be missing some folders.
I'm unable to access the site from my location for what ever reason so I can grab using FileZila.
I'm uploading what I have to my server in an attempt to get something working but I still have over 4 gigs to go.
Title: Re: Resurecting x10's forum?
Post by: Jeff on August 05, 2013, 02:12:37 PM
Thanks to the efforts of Tuicemen and the folks at Authinx, the Powerline Control Reliability board is back on line:

    http://forums.x10.com/index.php?board=102.0

Jeff
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 06, 2013, 05:27:41 AM
There are a couple of files that cause upload error these may have been what cause Jeff's file uploads to fail.
FileZile just reports these files and continues on.
I still have over 1.5 gigs in queue once it is finished I'll see what the problem files are and retry them.
I have a few tricks to try to enable this to be resurrected but there are source files that are missing.
A update or reinstall of SMF may help with that ???
I've yet to hear anything from my X10 IT contact.
Title: Re: Resurecting x10's forum?
Post by: Jeff on August 06, 2013, 07:02:12 AM
There are a couple of files that cause upload error these may have been what cause Jeff's file uploads to fail.

In my case I believe it was the sheer size of the upload.  Reading the terms of service again in detail, they say they do "NOT provide unlimited space for online storage, backups, or archiving of electronic files, documents, log files, etc".  And they do mention 2GB as an upper limit for a database.  Going beyond that can result in additional fees or termination of service.

The next step up in service would provide sufficient capacity, but it would increase my cost by a factor of four.  I would be willing to do that if the forum could be reconstructed to the point that it would be active again, and not just a dead archive.

The compressed file is available if someone wants to recreate the archive on their own computer.

Jeff
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 06, 2013, 04:11:37 PM
I finally got all files uploaded to my server.
However besides it missing several folders it is missing crucial sub folders and files from the folders that were copied.

I've attempted a rebuild using 1.1.4 version install files  (the  version the forum was running)  as well as attempting a repair settings.
Both attempts have failed but I'm getting write errors. The file attributes need to be set by my server host as my attempts have failed :'(
I'm doubtful this will allow a rebuild or repair at this point but there is nothing to lose right now. ::)

I've also requested the x10 forum files or for it to be reactivated however I doubt anyone (real person) will reply.
Title: Re: Resurecting x10's forum?
Post by: Jeff on August 06, 2013, 05:56:07 PM
I finally got all files uploaded to my server.
However besides it missing several folders it is missing crucial sub folders and files from the folders that were copied.

When I unpacked the files on my PC, I was able to enter at the top and then select any of the individual boards and topics in them.  They are all strictly read-only, but the structure appears to be complete except for the search tool.  And obviously, all the links that went to the previous X10 website are now dead.

I think that if I was able to upload all 120,000 files to my server, it would have worked the same as it does here on my PC.  You can check the one board that I did upload:

    http://jvde.us/x10_forum_pcr/index.php@board=102.0.html

I understand this is just the html file structure, and it is not running on the forum software.

Jeff
Title: Re: Resurecting x10's forum?
Post by: mldenison on August 06, 2013, 06:16:09 PM
Pretty neat, Jeff.  Good work.
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 07, 2013, 06:37:22 AM

I think that if I was able to upload all 120,000 files to my server, it would have worked the same as it does here on my PC.  You can check the one board that I did upload:

    http://jvde.us/x10_forum_pcr/index.php@board=102.0.html

I understand this is just the html file structure, and it is not running on the forum software.

Jeff
Don't get me wrong, I do have the forum setup using the html files structure.
However there were files that lead me to believe this could be more then just a data base.
I was hoping to have something other then that similar to The Way back machine  (http://web.archive.org/web/20080828232306/http://www.x10community.com/forums/)
I'm cleaning up the extras I added trying to get a working forum but at least we have a read only data base which stopped being active July 29, 2013, 12:44:34 PM  :)
I'll try to keep this up until hopefully someone reactivates the forum
The X10 Forms (http://tuicemen.com/x10forums)
Title: Re: Resurecting x10's forum?
Post by: Jeff on August 07, 2013, 06:59:19 AM
Quote
Don't get me wrong, I do have the forum setup using the html files structure.
However there were files that lead me to believe this could be more then just a data base.
I was hoping to have something other then that similar to The Way back machine  (http://web.archive.org/web/20080828232306/http://www.x10community.com/forums/)
I'm cleaning up the extras I added trying to get a working forum but at least we have a read only data base which stopped being active July 29, 2013, 12:44:34 PM  :)
I'll try to keep this up until hopefully someone reactivates the forum
The X10 Forms (http://tuicemen.com/x10forums)

That's GREAT!  It is exactly what I was trying to do myself until I ran into the size limitations.

Jeff
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 07, 2013, 08:14:18 AM
There still is a lot of unnecessary files that can be removed to free up some space.
I'll slowly get them removed.
I only wish (still hope) a working X10forum could/can be recreated. ???
Title: Re: Resurecting x10's forum?
Post by: woodyl on August 07, 2013, 11:51:41 AM
Tuicemen:  Good work!  Looks like the old forum is working pretty well.
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 07, 2013, 12:42:57 PM
Thanks, at least we didn't loose the valuable info it contained.
Not to take any thing away from this forum, I've setup sections on my forum to continue on if users experience software issues not covered it the old forum.
Title: Re: Resurecting x10's forum?
Post by: rsisinni on August 08, 2013, 08:15:03 AM

I'm going to attempt to parse the html and make a live forum out of it.

   There will still be some problems, first of all people will need to claim their accounts, I do not have email addresses so there needs to be some kind of process defined on how to do this.  Hopefully an automated process.

As for hosting It, I was thinking that some kind of foundation or club can be formed to keep it going, that way it won't be in the hands of one person and if they lose interest, get hit by a car etc,  it won't disappear.

If I manage to achieve an import, the forum size should become a lot smaller, and more manageable.



Some other updates:
1)  I am doing another backup of the site this time with attachments, I managed have wget login before doing the backup, so this time the attachments should come down, it will probably take a couple of days to complete.
2) I have backed up the knowledge base
3) I have done a complete backup of the ftp site, (previously I had only retrieved the manuals).


Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 08, 2013, 09:56:30 AM
If you manage a import you'll need to get the repair_settings.php file from SMF so that you can gain access to change the forums file locations.

Once that is done a update to the current version it may allow extra automatic options not found in 1.1.4.
In any case you'd need Admin access. I can help with that.
If your successful using the repair_settings.php file all users will have access with out any extra work.
Title: Re: Resurecting x10's forum?
Post by: rsisinni on August 08, 2013, 06:27:15 PM
Hi Tucemen, since I'm working the web pages that I grabbed and not the database, since I don't have access to it. What I am doing is parsing the pages, and extracting as much information as I can from it.  From the extracted information I will directly write into the database, reconstructing a version of the forums.   I have made some progress  I'm extracting and am pulling out plain text posts, but I want to try to keep some formatting.  I need to learn a little more about the database structure of the SMF forum, I don't know if they put HTML directly into the database for things like that.  Anyway I'm setting up a sandbox SMF forum so I can post into it and see what happens in the database.
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 20, 2013, 04:55:20 PM
Some good news.
I've been able to finally get on the X10forums and backup the file structure and do a complete database backup.
Though I've never tried to restore a forum from a SLQ back up I'm going to give it an attempt.
If anyone is experienced in this I'd love to hear from you. ;)
Title: Re: Resurecting x10's forum?
Post by: Jeff on August 20, 2013, 09:12:14 PM
That is really good news.  While you have already preserved that storehouse of information, having a complete working Forum again is certainly better.

On another news front, I'm pursuing an alternate X10 controller that I discovered last week.  One is on order for evaluation, and I hope to post the results here in a week or two.

Heff
Title: Re: Resurecting x10's forum?
Post by: hawk1 on August 21, 2013, 04:19:05 AM
Thank God for you guys!   :)    8)
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 22, 2013, 07:13:59 AM
I've submitted a support ticket to Godaddy about this.
It seems the x10forums SQL backup is to big for the MySQL download restrictions currently in place.
Hopefully they will increase the limit to allow me to upload or better yet do the whole rebuild thing for me!  8)
 ;)
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 23, 2013, 05:03:52 AM
Well news from support wasn't what I'd hoped. :'(
They won't increase the upload limit for me .
Files need to be 2meg or smaller so the data base needs to be broken up into 2meg or smaller chunks for uploading.
Since I have no experience with SQL this is going to take some time to figure the whole think out. :(
However I haven't given up.
I'm still looking for anyone experienced with forum restoring and or SQL.
Title: Re: Resurecting x10's forum?
Post by: rsisinni on August 24, 2013, 05:03:13 AM
Do you have ssh access, if so you can compress the database before you download it.
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 24, 2013, 05:49:31 AM
File is 32 meg compressed and support isn't willing to hep with this matter. :'(
However I downloaded SQL Workbench and was able to upload the complete backup to a newly created data base.
However I'm learning this is only a part of the puzzle and this will be a long proccess.
If I could get  the files as well as the data base it would be so much simpler. ;)
All attachments will be lost which in some cases were what made the post. :'(
It may be possible to retrieve some of these manually however that too would be a big job.
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on August 29, 2013, 04:52:21 AM
There is a bit of good news on this front.
Although I'm still fighting this one there are others that have taken up the fight.
Some have reported a bit of progress but it is still a long go.
User with experience in SQL will most likely have better luck.
As would someone that has restored a forum in the past.
But even with that experience it was a very large forum and will take some time. ;)

I've been able to pull the backup into a created forum however I'm still missing a few steps. ??? :-[
The new forum sees the boards the number of posts and numbers of threads however not the members nor the actual posts  ???
Title: Re: Resurecting x10's forum?
Post by: bkenobi on December 16, 2013, 10:49:14 AM
The old X10 forums are back up today!  I have not tried posting, but I was able to load the main page and at least one thread.
Title: Re: Resurecting x10's forum?
Post by: Tuicemen on December 16, 2013, 11:38:57 AM
Yes, sorry  I didn't get that posted here in this thread. I've been busy restructuring the old forum.
It is up and running, there is a thread on my forum which sheds some more light on things
http://tuicemen.com/forum/index.php?topic=688.0