Jump to content
C.M

[Requesting peer-review] BOINC project whitelist guidelines/checklist

Recommended Posts

C.M

In the past, we've used the following guidelines/questions when evaluating projects for whitelist worthiness, I'd appreciate a peer review to improve the quality/usefulness of the list:

 

  • Provide a brief description of the BOINC project.
  • Are there any planned/published scientific papers related to the BOINC project?
  • Is the team 'gridcoin' team founder position held by a known moderator/trusted individual?
  • Link to TOC, privacy statements, are the links present on their site?
  • Are there any software dependencies? VirtualBox, Docker, etc.
  • What software/hardware is the project compatible with? (CPUs, GPUs, 32/64bit, OS, etc).
  • Provide contact details of project admin (email or forum account for private messaging).
  • What is the category of the project? (Volunteer/non-profit | Commercial/For-Profit | BOINC DAC (BDAC) ).
  • If the BOINC project is commercial/for-profit:
    • Does the project intend to reward BOINC users for their contributed work?
      • Rewarding users with GRC or self-issued token?
    • Is the project backed by a company?
  • Is the project suitable for Proof-Of-Research? (Is work verifiable? Non-fake-able results)
  • Does the project have an entry on netsoft/boincstats/free-dc/other BOINC statistics site?
  • Does the project website provide SSL encryption?
  • Does the project website utilize captchas for improved security?
  • Has the project secured itself from spam account creation?
  • How frequently does the project update their statistics extract files? (user.xml.gz host.xml.gz team.xml.gz etc).
  • Will the project have sufficient work units for team Gridcoin to continuously crunch? More than a month of inactivity will lead to probable invalidation of whitelist status.
    • Is there a delay in the validation of returned results?
  • Have the project admins/owners acknowledged the existence of Gridcoin? Is the proposal to whitelist the project originating from the project administrator or a community member?
    • Will the project administrators issue a news 'notice' regarding their project being whitelisted in the Gridcoin network?
  • Is the project's purpose legal? Will the project have to exclude countries due to regulatory issues?
  • Is the project (and/or the project's distributed application) open-source?
  • Has the project implemented additional counter-collusion/counter-cheating mechanisms?
  • Upon virus scanning the work units being distributed by the project, are there any detections? (https://www.virustotal.com/ being the most reliable).
  • Does the project publicly accept Gridcoin donations?

 

Any other suggestions?

Edited by C.M (see edit history)
  • Like 3

Share this post


Link to post
Share on other sites
Gunde [closed]

Could add one something like this :

 

Is there an optimized application from community that is not implemented in your application? If so. Are there any plan to add the code in your application?

 

As this is could be hard for users to see but project admin would notice. There have been a few external application that users have been able to use could make ha huge change in credit rewards, so to inform users would be great.

Edited by 47an (see edit history)
  • Like 2

Share this post


Link to post
Share on other sites
Bernard

Ok, can we indicate what is a MUST and CAN criteria (Thanks Erkan_Yilmaz). For example is daily stats a MUST or CAN. CPDN switched to weekly stats as daily took too much resources. Secondly most of CPDN WUs take few weeks to complete and the current deadline for WUs is set to one year not to lock resources away from other BOINC projects. So some WUs are in progress for quite some time while the WUs hopper may be empty for over a month. Is this a huge problem, small or not at all. Perhaps finally every project can have a whitelist checklist card, so everyone can see how a project satisfies certain criteria. I guess this may be useful in voting for informational purposes.

Share this post


Link to post
Share on other sites
C.M

Details for Gridcoin Poll Poll On New BOINC Project Whitelist Mandatory Requirement - Stats Exports. Created 2018-10-06 14:42:40; Ends 2019-01-10; Running for 3 months, 4 days

 

Correction: NOT Universe AT Home, rather Einstein AT Home!

Should BOINC projects maintain publicly available BOINC statistics dumps as a mandatory BOINC whitelist requirement? Soon the quantity of people allowed to scrape [email protected] and Primegrid stats will drop significantly. There are plenty of projects which could replace these projects on the whtielist.

 

Edited by C.M (see edit history)
  • Downvote 1

Share this post


Link to post
Share on other sites
limacoin

This is one of the more stupid votes of the GRIDCOIN community! I would say one, which is described best by: “shoot yourself in the foot!”

The BOINC projects restricting the access to their data- and user-base comply with the EU Data Protection Directive (May 25, 2018), so to cast a vote: “Should it be mandatory that whitelisted BOINC projects produce statistics exports for the public without authentication?” is rather senseless at best or askes to break to law at worst.

I quick check of the different locations or institutions supporting the whitelisted project shows that around 11 projects are based in Europe:

1.       [email protected]

2.       Distributed Hardware Evolution Project

3.       GPUGRID.net

4.       [email protected]

5.       ODLK1

6.       PrimeGrid

7.       TN-Grid

8.       Universeathome

9.       VGTU [email protected]

10.   YAFU

11.   [email protected]

So if this vote goes through, around a third of all whitelisted projects will be potentially delisted.

@C.M: Please illuminate us, which projects comply with our standards and are not whitelisted yet, as you indicate: “There are plenty of projects which could replace these projects on the whitelist.”

Share this post


Link to post
Share on other sites
C.M
On 10/10/2018 at 9:46 PM, limacoin said:

This is one of the more stupid votes of the GRIDCOIN community! I would say one, which is described best by: “shoot yourself in the foot!”

I disagree, there have been worse polls on the Gridcoin network in the past - we voted to not require SSL as a mandatory requirement in the past since 8+ projects at the time didn't have SSL implemented, that was perceived similarly because it affected so many projects, yet it was a severe security risk which likely leaked thousands of BOINC users credentials to multiple MITM attackers.

 

I perceive the restriction of access to these stats files as a centralization risk and until we've got DWP implemented I'd rather see a dozen small BOINC projects replace BOINC projects which chose to cut us off with minimal notice despite being one of their largest sources of compute power.

 

On 10/10/2018 at 9:46 PM, limacoin said:

The BOINC projects restricting the access to their data- and user-base comply with the EU Data Protection Directive (May 25, 2018), so to cast a vote: “Should it be mandatory that whitelisted BOINC projects produce statistics exports for the public without authentication?” is rather senseless at best or askes to break to law at worst.

 

The projects which would be immediately affected are Einstein at Home (@ gets withdrawn?) and PrimeGrid since they have restricted access and implemented authentication mechanisms to access exported stats dumps, no other European BOINC project at this stage has followed in their footsteps. PrimeGrid's justification was the severe bandwidth drain that distributed downloading by external systems places on their web servers rather than being scared of EU regulations.

 

For bandwidth usage why not pick a different data serialization method than XML like msgpack, json or protobuf3 and filter inactive accounts like:


Gridcoin Research HUG REST API. Contribute to gridcoin-community/GRC-HUG-REST-API development by creating an account on GitHub.

 

On our end we could eliminate distributed data downloads through TomasBrod's "Dynamic Withess Participation proposal" in the future (not within the 7 day notice primegrid provided)


Gridcoin-Research. Contribute to gridcoin-community/Gridcoin-Research development by creating an account on GitHub.

 

IMO, Gridcoin will never be a GDPR compliant cryptocurrency, given that current user registration data (and old beacon email data) cannot be purged from the blockchain without a new blockchain:


Gridcoin-Research. Contribute to gridcoin-community/Gridcoin-Research development by creating an account on GitHub.

 

This is technically a breach of "User Right-of-erasure - deleting a user's account" mentioned in the recent BOINC GDPR presentation: https://drive.google.com/file/d/1t2bGnxLks9V6pkkUwK0kQWXFpwMuFgIX/view

 

On 10/10/2018 at 9:46 PM, limacoin said:

I quick check of the different locations or institutions supporting the whitelisted project shows that around 11 projects are based in Europe:

1.       Asteroids at home

2.       Distributed Hardware Evolution Project

3.       GPUGRID.net

4.       LHC at home

5.       ODLK1

6.       PrimeGrid

7.       TN-Grid

8.       Universeathome

9.       VGTU project at Home

10.   YAFU

11.   yoyo at home

 

DHEP is located in the UK, soon to be out of the EU.

OLDK is Russian, so no GDPR concern there.

PrimeGrid & Universe at Home have both restricted access, so would be immediately affected, sure.

 

None of the other BOINC projects you mentioned have thus far indicated they are going to restrict access to these stats exports. GDPR compliance has been implemented in BOINC (if the projects have updated their BOINC server recently) so they may be content with not restricting access in the future. If they do restrict access in the future (and assuming this poll had passed) then they would be ineligible for the whitelist.

 

On 10/10/2018 at 9:46 PM, limacoin said:

@C.M: Please illuminate us, which projects comply with our standards and are not whitelisted yet, as you indicate: “There are plenty of projects which could replace these projects on the whitelist.”

 

Which standards are you referring to? Because as far as I'm aware the primary whitelist requirements are readily-available daily stats exports, sufficient & stable supply of compute tasks, a production-ready state not test/beta project, open registrations, non-cheatable tasks and a passed whitelist poll, it's not a high barrier to entry. There's plenty of community best-practice requirements but it's up to the BOINC project admin's discretion which to follow and what to disclose to the public.

 

Anyone can create a BOINC project and distribute whatever distributed computing task they desire to realize significantly cheaper and more diverse computing tasks compared to competitor DC crypto projects.

 

We aught to be encouraging the brainstorming and creation of new BOINC projects and helping bootstrap small BOINC projects rather than be fighting over projects which threaten to cut us off (despite our huge compute capacity) unless we implement centralized measures to accommodate the EU regulations they fear and which we breach.

 

Regarding which existing projects could apply for whitelist status the following from a quick glance from boincstats:

 

Acoustics at home
BURP (If they can sustain workload)
Climate Prediction (If they produce daily stat dumps)
DBN Upper Bound
DENIS at Home
Gerasim at Home
GoofyxGrid at Home (CPU, not NCI)
MindModeling at Home
Moo! Wrapper (Voted out recently due to volunteers not liking topic of research, otherwise valid)
Primaboinca
RADIOACTIVE at HOME (if geigercounters are publicly available & not cheatable)
RakeSearch
RNA World (If they get the requested funding donations to survive)
WEP-M+2 Project (if not 32bit only still)
XANSONS for COD (edit: Not viable, completed project)

Edited by C.M
copy/paste errors & project name @ fix v2 & grammar (see edit history)
  • Like 2

Share this post


Link to post
Share on other sites
Michael Goetz

Dear C.M,

 

I'd like to clear up a couple of misconceptions.  It's always good to get the facts straight from the horse's mouth, so to speak.  I'm one of PrimeGrid's administrators, and I am the person most directly responsible for restricting access to the statistics. 

 

We did not do this with careless disregard for Gridcoin.  It was directly targeted at Gridcoin.

 

But not for the reasons you might think.  There was no animus involved.

 

In early September I noticed that PrimeGrid was virtually unreachable for a few minutes.  It turns out there was a DDOS attack against us, but it was a short lived attack.  It only lasted a few minutes.  Some investigation determined that this had been happening about once a day for a while, but since it was short in duration nobody really noticed.

 

By now you probably have guessed where this story is going.

 

Gridcoin needs our statistics, and collects them by having a large number of host computers each collect our stats files.  At pretty much the exact same time.  And these are really large files.  While we have a fairly hefty server, and a decent pipe to the Internet, Gridcoin *is* a significant player at PrimeGrid, and all those hosts (over a thousand) downloading those large files at the same time was completely saturating our Internet connection.

 

PrimeGrid isn't the largest BOINC project, but we're up there, and our servers are more powerful than most.  I'm not saying that to brag.  The point is that if Gridcoin can effectively, albeit unintentionally, mount a successful DDOS attack against PrimeGrid's servers, imagine what Gridcoin is doing to all the other BOINC sites with much smaller hardware.  Their only saving grace is that their stats files are almost certainly smaller than ours, and they may have fewer Gridcoiners (is that a word?) on their system.  Nevertheless, this method of gathering the statistics is undoubtedly affecting many BOINC sites adversely.  What is certain is that it has been making PrimeGrid's server unreachable for periods of time.  That's unacceptable.  Except for the fact that this is clearly unintentional, it would literally be criminal.

 

Back in September I reached out to Gridcoin.  There's a fix in the pipeline.  Things will hopefully be back to normal soon, minus the DDOS part.

 

We're not the bad guys here.  Nor is Gridcoin.  Everyone is working together to get this working.

 

Best wishes,

Michael

 

P.S.  I believe you should take XANSONS FOR COD off your list.  That's not a viable project for Gridcoin.  Xansons' work was completed about a year ago.  Occasionally there's a tiny bit of work they run, but the WUs only last an hour or two.   They also restrict the ability of new users to join the project.  As I understand it, the lack of work and new user restrictions make them ineligible for whitelisting, correct?

Edited by Michael Goetz
Spelling (see edit history)
  • Like 5

Share this post


Link to post
Share on other sites
dblanch256

@Michael Goetz quoted:

 

"Gridcoin needs our statistics, and collects them by having a large number of host computers each collect our stats files.  At pretty much the exact same time.  And these are really large files.  While we have a fairly hefty server, and a decent pipe to the Internet, Gridcoin *is* a significant player at PrimeGrid, and all those hosts (over a thousand) downloading those large files at the same time was completely saturating our Internet connection."

 

I am not a Gridcoin expert, but I found your post to be very informative.  Given the issues you described, it would appear that several mitigation paths might be possible.  For all I know, some are being explored as we speak.  Clearly Gridcoin must gather statistics.  But are we taking more than we need?  Must we take it in a single daily burst?  If some kind of throttling would help, on whose end should it logically be?

 

My personal experience with PrimeGrid has been uniformly positive.  Compared to other whitelisted projects, I would rate it one of the friendliest.  You always have available work units, the system is rarely ever down, and both CPU and GPU tasks run very cleanly on my hardware.  [I wish I could say the same for other projects.]

 

I really hope this is just a hiccup.  I have always considered BOINC to be a disruptive technology and having studied its architecture I am even more convinced that it is the prevailing "best of breed".  It is so much more than simply a "distribute and collect" service.  Also, something else you mentioned rang a bell ...

 

When I was working for the FAA, we had a similar issue with too many user communities requesting data from the Host system (which processes all filed flight plans).  Once we realized that these requests were "similar but different" we added a HADDS component as the only data collection client of the HOST, and whose job it was to periodically draw a "union" of the requested data, filter and disseminate it to everyone who had signed up for customized Host data.  This simple trick unloaded the Host by an order of magnitude or so.  [Whether this approach is applicable to your project, I don't know.]

 

In summary, I'd simply like to see PrimeGrid survive for use by Gridcoin, for all of the above-mentioned reasons!

Share this post


Link to post
Share on other sites
Michael Goetz
9 hours ago, dblanch256 said:

I really hope this is just a hiccup.

 

It should be just a hiccup.  I expect it to be fixed soon, if it's not already.  (The fix is on the Gridcoin side.)

Edited by Michael Goetz (see edit history)
  • Thanks 1

Share this post


Link to post
Share on other sites
Michael Goetz
20 hours ago, dblanch256 said:

I am not a Gridcoin expert, but I found your post to be very informative.  Given the issues you described, it would appear that several mitigation paths might be possible.  For all I know, some are being explored as we speak.  Clearly Gridcoin must gather statistics.  But are we taking more than we need?  Must we take it in a single daily burst?  If some kind of throttling would help, on whose end should it logically be?

 

The problem was that Gridcoin was grabbing a thousand copies of those files, all at once.  

  • Like 1

Share this post


Link to post
Share on other sites
jamescowens

The issue we are grappling with is that because Gridcoin operates in a decentralized manner, a subset of nodes (25% of the nodes running the "neural network") independently pull the statistics files. This is normally mitigated by a form of a proxy server to pull the stats files, but that has been subject to availability issues as we make changes to support authenticated stats access. The nodes will first try to use the "proxy", but if they fail there, they will fall back on direct access. We have repaired the "proxy", which we call the "scraper", to use the authentication provided by PrimeGrid, so for the issue should be alleviated shortly.

 

We are going to set up additional instances of the scraper for RAS to reduce the chances for the nodes to fallback on entire stats downloads.

 

@Michael Goetz If we could get SHA-256 or even better SHA-512 hashes of the statistics files to confirm the file authenticity and version, this would further reduce the load on your servers.

  • Like 3

Share this post


Link to post
Share on other sites
Michael Goetz
30 minutes ago, jamescowens said:

 

@Michael Goetz If we could get SHA-256 or even better SHA-512 hashes of the statistics files to confirm the file authenticity and version, this would further reduce the load on your servers.

 

I don't see how that would help, but I have only a rudimentary understanding, at best, of how your system works..  Presumably your nodes trust your proxy/scraper.  

 

 

Share this post


Link to post
Share on other sites
jamescowens
52 minutes ago, Michael Goetz said:

 

I don't see how that would help, but I have only a rudimentary understanding, at best, of how your system works..  Presumably your nodes trust your proxy/scraper.  

 

 

 

They do now, but that is not ideal. The whole concept of blockchain involves nodes independently applying the rules and interacting with other nodes to achieve consensus. The idea of the nodes blindly "trusting" the proxy is tolerated right now, but in tension with how a blockchain operates.

 

Remember we are actually minting and distributing rewards that have monetary value based on the statistics, so this is not just an exercise in reporting. Without verification we have to consider the possibility that someone could be motivated to try a man in the middle attack on the proxy itself.

 

Here is how the hash verification files help... The large files are hashed by you and both the files and the hashes themselves are stored for pickup. The proxy downloads all of the files once.

 

The nodes download the files from the proxy, then execute their own hash. They download *ONLY* the hash files from your BOINC server and compare the hashes from that direct download to the calculated hashes from the proxy download. If they match, there is a VERY HIGH degree of confidence that the files on the proxy have not been tampered with.

 

You will still have many nodes talking to your stats server, but they only will be requesting the very small hash files, not the gigantic stats files.

 

This would be a good compromise to balance the bandwidth requirements with our requirements that their be a direct trusted relationship between the node and your BOINC stats site.

 

(And the hash files should not be protected by authentication, since they contain no PII that would be subject to GDPR or other privacy laws, so this allows us to maintain having only the proxy "login" to get the stats.)

Edited by jamescowens (see edit history)
  • Like 2

Share this post


Link to post
Share on other sites
IKI

Thanks for the infos Michael, I was kind of wondering what was happening. All clear now!

Share this post


Link to post
Share on other sites
IKI

I also, like DBLANCH256, must say that my experience with PrimeGrid has always being absolutely spotless, and I am there since a while, way before joining Gridcoin...

Very professional, reliable, loads of interesting sub-projects, knowledgeable and very active community, if you are into it regular challenges, etc..

To sum it up this is one of the top projects on Boinc and thus it would be a shame if it would end up not being whitelisted anymore.

 

But fortunately it seems like, if I understand the issue correctly, that a fix is not such a big deal to implement.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Cryptocurrenytalk Logo

 

News, information, and discussions about cryptocurrencies, blockchains, technology, and events. Blockchaintalk is your source for advice on what to mine, technical details, new launch announcements, and advice from trusted members of the community. Cryptocurrencytalk is your source for everything crypto. We love discussing the world of cryptocurrencies.

 

   
×
×
  • Create New...

Important Information

By using CRYPTOCURRENCYTALK.COM, you agree to our Terms of Use.