ORG-tech-vols IRC meeting log 2014 02 25
New page
= ORG-tech-vols IRC meeting log 2014 02 25 =
<pre>
19:31 < graphiclunarkid> OK - I think it's that time so let's get going.
19:31 < dantheta> I keep trying to get the hang of doge stuff, but I was born to lolcat
19:32 < _46bit> graphiclunarkid: ah good, wasn't sure which clock was rught
19:32 < graphiclunarkid> Isn't it great how the internet has produced two independent grammatically-incorrect memes around differentn pets ;)
19:33 < dantheta> I did read a paper describing the greater internal consistency of doge (vs. lolcat), but I run the risk of going very off-topic.
19:33 < graphiclunarkid> So... I can haz progress reports nao?! Wow, so contribution, many community, such website. Amaze!
19:33 * graphiclunarkid waits to be kickbanned....
19:33 < dantheta> I'm pretty impressed ...
19:33 < korikisulda> You can haz progress reports. Much development, such progress, very success, so coding
19:34 < graphiclunarkid> :)
19:34 < dantheta> Much framework, so database, very results.
19:34 < dantheta> (I'm still trying ... :S)
19:34 < korikisulda> I basically just started working on getting the client to work with the new version, so that's fun
19:34 < graphiclunarkid> I've seen a whole bunch of code fly up onto the github repos this week - which is nice!
19:34 < Allll> I got the basic HTML up on github, but haven't got much further. I'll hopefully start looking at the staging server this week (end)
19:35 < graphiclunarkid> korikisulda: Cool! Is that the android client?
19:35 < korikisulda> Sorta
19:35 < korikisulda> It's the Java library
19:35 < korikisulda> Which is what I mean by client
19:35 < korikisulda> Sorry if I got your hopes up ;D
19:35 < graphiclunarkid> korikisulda: Are we planning to refactor the java client to use the library you've written? That would seem sensible to me but I don't know what Gareth thinks.
19:36 < korikisulda> I don't know yet
19:36 < dantheta> I haven't heard anything from him in a while ...
19:36 < korikisulda> My android development skills are pretty terrible
19:37 < graphiclunarkid> dantheta: yeah, not sure what he's up to. I was planning to drop him a line to see how he's doing.
19:38 < graphiclunarkid> korikisulda: I guess getting the java library talking to the new API dantheta's been working on isn't going to be slowed up by the android client stuff - so I guess just crack on with that :)
19:38 < dantheta> DO we still want to use the google cloud messaging bits in the java api?
19:38 < graphiclunarkid> The android client will have to change anyway to use the v1.2 API - it'll be easier to do that if there's a lib we can drop in already!
19:39 < dantheta> I haven't really prodded at those much on the server side.
19:39 < graphiclunarkid> dantheta: I don't know - what do you think?
19:40 < dantheta> I'm unsure - Gareth mentioned at the december meetup that it was a power&bandwidth optimisation useful for android devices, but other clients should just poll.
19:41 < graphiclunarkid> Well, since we have an android client, mybe it's worth continuing to support the feature?
19:42 < dantheta> korikisulda: Is the GCM stuff useful for java in general, or would you want to keep its usage specific to android? I'm happy to support it either way, I just don't really understand it too well at the moment
19:42 < korikisulda> I don't know how we could use GCM unless we're on Android
19:42 < korikisulda> I'm pretty sure it is specific to that
19:43 < korikisulda> Either way, it's not difficult to support in terms of contacting the API, so I have no problem with it
19:43 < dantheta> hehe, the extent of my unknowingness of GCM is revealed! I'll port the existing code to the new framework and signature stuff, and hope it's useful later.
19:44 < korikisulda> Oh, believe me, I know nothing about it either
19:45 < graphiclunarkid> Allll: I saw you pushed a bunch of assets to the web repository on github :)
19:45 < korikisulda> I didn't expect setting parameters for get requests to be so painful *curses apache*
19:45 < korikisulda> I might just use the cheaty option and append them to the URL ^.^
19:45 < dantheta> Rewrite rules?
19:46 < graphiclunarkid> Allll: Did you see my email to the list last night about integrating that stuff with the staging server?
19:46 < korikisulda> I'm referring to apache commons in this case
19:46 < dantheta> Oh right :)
19:46 < Allll> yeah, that's the basic HTML for the site, it's bootstrap so should be pretty responsive, but if anyone finds any bugs on any devices let me know
19:46 < korikisulda> Which now I come to think about it, is an amazing way to get confused..
19:47 < graphiclunarkid> Allll: Did I miss it being hosted somewhere? Sorry, been a bit scatterbrained with emails this week!
19:47 < Allll> yeah, saw that never having used modx doesn't help but I can't see a practical way to put it in version control
19:47 < Allll> no, it's standalone HTML so justs unzip and take a look
19:47 < graphiclunarkid> Allll: Yeah, me either, without doing a lot of jumping through hoops.
19:47 < graphiclunarkid> Allll: Ah, ok, yep. Will do.
19:48 < Allll> I need to go over it again, but I think it's enough to start chopping up and putting into modx templates
19:48 < graphiclunarkid> Looking at it now - looks pretty good!
19:49 < Allll> that reminds me it still needs a name!
19:49 < korikisulda> Hadrian's house?
19:49 < graphiclunarkid> I think you're right - we probably need to do the implementation in MODX directly rather than being clever about it with git.
19:49 < graphiclunarkid> Git is cleverer than I am - I have proven this on a few previous occasions. It is definitely not afraid to kick me at the worst possible moment!
19:50 < Allll> Can you automate some kind of regular dump of the server/db at least that way if we experience some epic fail we can kind of roll back
19:50 < Allll> I know the feeling!
19:51 < graphiclunarkid> Probably, yeah.
19:51 < graphiclunarkid> There are plugins that let you treat elements as static files, e.g. http://modx.com/extras/package/staticsaver, but I think that might be too much effort!
19:51 < Allll> 46bit - do you reckon you'd be able to look at that JS stuff I mentioned in mailing list? I know it was quite a bit!
19:52 < Allll> regular dumps would keep me happy (oo-er)
19:52 * graphiclunarkid sniggers
19:52 < _46bit> Allll: I'm doing some of it tonight
19:53 < Allll> sweet - I wasn't looking forward to that! but let me know if there's anything you need from
19:53 < graphiclunarkid> Allll: Um, I think I've spotted a bug, sorry!
19:53 < Allll> no worries, just let me know I'll get it all fixed
19:54 < graphiclunarkid> If you reduce the browser width to the point where the PROJECT NAME title overlaps with the menu, but not so far that the menu becomes vertical, then hovering over the "check a site" or "unblock a site" menu options superposes a white box on top of part of the PROJECT NAME text.
19:54 < graphiclunarkid> I'll raise an issue on github.
19:56 < dantheta> There was an offlist enquiry about code reuse in the API, and possible adoption of a REST framework for writing it in. After a brief request for recommendations on the list, I've had a pleasant time using Silex. I'm ready to merge that, if everyone's happy?
19:57 < korikisulda> I've had no problems with it
19:58 < dantheta> Cool. I've also been looking at the API's handling of the urls-to-test queue.
19:58 < graphiclunarkid> Allll: Issue raised :)
19:58 < Allll> yay, now to fix it!
19:58 < graphiclunarkid> Allll: That's one reason to keep using github, of course. Issue integration is very nice...
19:59 < Allll> yeah I'll keep an eye out and work through anything that comes up
19:59 < _46bit> So in terms of the frontend All, did you build it from some wireframes?
20:00 < _46bit> Oops
20:00 < korikisulda> Time travel, awaaaaaay!
20:01 < Allll> I did the mockups based on the bootstrap grid, I tend to use that as it saves a lot of time reinventing the wheel, and I used HTML boilerplate as a starting point
20:01 < Allll> html5 boilerplate that is
20:02 < Allll> I customised both bootstrap and boilerplate to remove stuff I didn't expect to use so if something doesn't work as expected try to use the full bootstrap CSS/JS files
20:03 < graphiclunarkid> Allll: Maybe the thing to do WRT github is to continue iterating there for now, just viewing the static files, and raising / closing issues as we go. Once we're happy with everything we can tag up a release and then chop it up and implement it as MODX templates (really shouldn't take too long from what I can see).
20:03 < _46bit> When I get back to my Mac I'll have to take a look at the wireframes
20:03 < graphiclunarkid> We can then keep going along those lines: flesh out changes in github, make sure everyone's happy, implement in MODX.
20:04 < Allll> Yeah I'm happy to do that for next few days until we've had a good testing and resolved any issues, I don't want to put off getting into modx too long though
20:05 < Allll> If you have any questions just drop me an email _46bit
20:05 < graphiclunarkid> Allll: Agreed. I will try to encourage some more comprehensive review and comment - starting by doing so myself!
20:05 < graphiclunarkid> dantheta: How close are you to needing to get your v1.2 API hosted on the server for testing?
20:06 < dantheta> It's pretty close. I've got some queueing issues to take care of in the code, but this version of 1.2 is pretty solid. All I'd need is access to the server!
20:06 < korikisulda> It'd mean I wouldn't need to host it on korikisulda.net any more :D
20:07 < Allll> bug fixed and pushed :D
20:07 * korikisulda dances
20:07 < graphiclunarkid> dantheta Excellent. I will arrange for you to have access in that case.
20:07 < dantheta> Now that 1.2 can take URL requests and record results, it's probably time for the DB to have a more permanent home
20:08 < graphiclunarkid> Allll: Awesome - that was quick!
20:08 < graphiclunarkid> dantheta: Where is the DB living now?
20:08 < dantheta> api.bowdlerize.co.uk - although I don't know where that is
20:08 < dantheta> Possibly an ORG bytemark VM, I'm not sure
20:09 < graphiclunarkid> I suspect that's on the same VM as the staging MODX server.
20:09 < dantheta> Yep, same IP
20:10 < graphiclunarkid> I'll see if I can get api.blocked.org.uk pointed to that IP too. The database can then stay where it is :)
20:10 < dantheta> Yep. The 1.2 code does make a few DB changes, but they're mostly backwards compatible.
20:11 < graphiclunarkid> How does one access the api? Something like http://api.blocked.org.uk/rest/1.2/endpoint/action
20:11 < korikisulda> At the moment, using the setup I have, a valid URL would be "http://korikisulda.net/api/1.2/request/httpt"
20:12 < graphiclunarkid> Ah right.
20:12 < korikisulda> I imagine if it were set up correctly, it'd just be api.whatever.whatever/1.2/etc
20:12 < dantheta> Either option is available.
20:13 < dantheta> Probably example.com/<version> would be nicer though
20:13 < graphiclunarkid> No point saying "api" twice I guess.
20:13 < graphiclunarkid> Yeah
20:13 < graphiclunarkid> Needs to be a different subdomain as blocked.org.uk points to a different server at the moment.
20:13 < dantheta> I think the version that's on bytemark at the moment has https too, subdomain is definitely best
20:14 < graphiclunarkid> Yeah, really should be TLS, not in the clear.
20:15 < vasilis> Any news regardings the Broadband connections at A&A?
20:16 < graphiclunarkid> vasilis: I haven't heard anything, no.
20:16 < graphiclunarkid> plett, you around?
20:17 < vasilis> Didn't do much last week but still found a way monitor the probes.
20:17 < graphiclunarkid> vasilis: Cool - do tell :)
20:19 < graphiclunarkid> vasilis: Also, have you had a chance to play around with the probe on my home connection at all? I confess I haven't looked at it since last week so by now you could have downloaded the whole internet and I wouldn't have noticed ;-)
20:20 < vasilis> Since the probes are going to respond via tor, I can use a monitoring tool such as munin or nagios to monitor the resources and health of the probes.
20:20 < dantheta> korikisulda: By the way, I think we'll need to make one more change to the /request/httpt endpoint, adding an ISP name to the request so it's /request/httpt/ispname.
20:20 < korikisulda> That would make sense
20:20 < korikisulda> How should we go about getting the ISP, though?
20:20 < vasilis> graphiclunarkid: Yes but it seems that your connection is not filtered ;)
20:21 < graphiclunarkid> vasilis: Well, I did warn you ;-)
20:21 < dantheta> I think it would have to be something the probe operator selects when they run the probe
20:21 < graphiclunarkid> vasilis: Using nagios would make sense - it would definitely be useful to get alerts if a probe were to go offline for some reason.
20:21 < vasilis> graphiclunarkid: True! But you helped me a lot with the hidden service.
20:21 < korikisulda> I think the Android probe uses http://wtfismyip.com/json, so I suppose that's an option :D
20:21 < dantheta> Ah yes, I'd forgotten about that
20:22 < vasilis> graphiclunarkid: As I said last week was a bit dead for me, not much acomplished...
20:22 < graphiclunarkid> korikisulda: dantheta: I meant to raise an issue against the android probe - I don't know whether it notices if you have a VPN active.
20:22 < korikisulda> I don't think it would, no
20:22 < graphiclunarkid> vasilis: Well, just let me know if you want me to do any more image testing. Happy to download and test new versions.
20:23 < dantheta> vasilis: hopefully I'll have some time to hook up the raspi and the filtered vodafone connection
20:23 < vasilis> graphiclunarkid: The good thing is that we have solved two main issues, thus how to monitor and access the probes.
20:23 < graphiclunarkid> vasilis: Indeed
20:24 < graphiclunarkid> I'm still thinking about taking the OONI RPi out for a trip to the local Starbucks and hooking it up to their wifi. That would get us some filtered-connection data for sure!
20:25 < graphiclunarkid> (I'd have to endure Starbucks coffee though...)
20:25 < vasilis> dantheta: I can give you a image link that you can just copy to it to an SD card then plug it in and boot raspi :)
20:25 < dantheta> korikisulda: The reason I'm thinking of adding the ISP to the queue request, is so that we can get a probe result for a newly-added URL from multiple ISPs as quickly as possible. It means appearing to run a queue per ISP, but that's doable
20:25 < dantheta> vasilis: That'd be cool!
20:25 < korikisulda> That makes sense, yeah
20:25 < graphiclunarkid> dantheta: can confirm vasilis's image works pretty much out of the box.
20:26 < dantheta> graphiclunarkid: Outstanding!
20:26 < vasilis> graphiclunarkid: with your valuable help of course!
20:26 < graphiclunarkid> However I did have to mount it on my laptop and add my own ssh key in order to access it myself.
20:27 < dantheta> The current (1.1) queue will send a URL out once, then cycle round the queue. It means we'd have to take <number of URLs in database> probe results before a URL gets a result for a second ISP
20:27 < graphiclunarkid> And I also had to regenerate the TOR hidden service key / URI, which were zero-byte files for some odd reason.
20:27 < graphiclunarkid> But other than that it was fine.
20:27 < vasilis> graphiclunarkid: This is fixed now!
20:27 < graphiclunarkid> dantheta I like the idea of separating the queue by ISP as long as we get consistent identification of ISPs.
20:28 < graphiclunarkid> vasilis: Glad to hear it :)
20:28 < vasilis> I still need the pub key thought
20:29 < graphiclunarkid> vasilis: For mass production we might want to consider allowing username/password ssh login to avoid that difficulty for random downloaders. Otherwise we'd have to get people to send us public keys and insert them into the images somehow - which sounds like a pain!
20:29 < dantheta> graphiclunarkid: Well, the ISP returned by the json wtfismyip API comes from a standardised list. Or for the interactive approach we could send a list of ISPs in the config file, which the probe UI could use to populate the list shown to the user on startup.
20:29 < korikisulda> The images wouldn't be compressed, though, right?
20:30 < vasilis> graphiclunarkid: True but I would like also to have 2 different images.
20:30 < graphiclunarkid> dantheta Indeed. The other obvious benefit of per-ISP queues is that there's no real point a probe polling for URLs that we don't want to test on the network to which it's connected. If we start targeting tests then that will become useful.
20:31 < dantheta> The other thing that crossed my mind with the android probe is if the phone changes connection (3G to WLAN), and how to make sure that the right ISP is still sent with any results.
20:31 < graphiclunarkid> dantheta: When people visit the site to report that a URL is blocked on their connection, rather than to find out on which networks it's blocked in general, there might be no need to test it elsewhere.
20:31 < vasilis> Support people that would like to run the Probe themselves without any monitoring/remote management for obvious reasons.
20:31 < Allll> I think android can tell if it's using data connection or wifi, so that could be checked to see if it's changed and if it does then trigger a recheck
20:32 < graphiclunarkid> vasilis: Yeah, two images would work for that.
20:32 < vasilis> korikisulda: We can have compressed and uncompressed images as well, or do I miss anything?
20:33 < dantheta> One other quick question about URL submission - are we accepting just root-level URLs (just domains, really), or will be accept full URLs pointing at specific pages on a site?
20:33 < korikisulda> Well, I'm just thinking that if the image is uncompressed, it'd be possible to insert the key quite easily.
20:34 < graphiclunarkid> dantheta: I think that depends on the subtlety of the mechanisms being used for blocking.
20:34 < graphiclunarkid> dantheta: At the moment they don't seem all that subtle!
20:34 < korikisulda> For DNS hijacking that would be best
20:34 < korikisulda> But if they do sneaky things with DPI, we'd have to check the individual URLs
20:34 < dantheta> Vodafone seems to use transparent proxy
20:35 < dantheta> But the result is the same
20:35 < korikisulda> As far as we're concerned, they're identical, yeah
20:37 < dantheta> OK, that's cool.
20:37 < dantheta> Just thinking about the database :)
20:37 < vasilis> So there is an X-VIA header option?
20:38 < vasilis> I meant VIA header.
20:38 < dantheta> I don't remember seeing one.
</pre>