Re: [cross-project-issues-dev] [epp-dev] [ide-dev] Phasing in Error Repo

[Markus Knauer <mknauer@xxxxxxxxxxxxxxxxx>]

Hi Marcel,

the good news is... with Luna such a change could not have been rolled out, with Mars I am very positive that we could do this. Although this is a possibility I don't like the idea a lot because a lot of work and risk is involved.

But now back to my thoughts:

- All Mars packages are configured in a way that they are looking for updates once a week; this makes it possible for the first time to roll out such a change but even in that case it is kind of a 'pull' by the user who needs to confirm a dialog here and there... it's not a push mechanism where we can trigger an update. And it is 'once a week' (starting Tuesdays) which could be too late?

- A change of the default preferences in packages is possible because all packages have a plugin_customization.ini of their own.

- As a reminder... from my perspective, but I don't know about all other projects, we have at least three bandwidth related risk factors in Mars, one of them is the error reporter, the other two are the aforementioned check-for-update and of course, the Oomph installer which both rely on p2 and the download server.

Marcel, if you want we can talk about options. If required I'll try to be available.

Thanks,

Markus (Knauer) ;-)

On 13 June 2015 at 08:58, Marcel Bruch <marcel.bruch@xxxxxxxxxxxxxx> wrote:

— Adding epp-dev to this discussion b/c the proposals below affect EPP packages —

The amount of traffic is something no one can estimate yet. I think everyone feels a little bit uncomfortable with this situation. So here is a proposal I’d like to discuss publicly (although only a small subset of people may feel need to discuss this).


Markus (Knauer),
in case we’d need to disable error reporting because of too much traffic: is it possible for you to simply rebuild the packages with an additional preference? This preference would preconfigure the system to “don’t send error reports".

How quickly could we do that - in worst case?


Denis,
since JEE is the largest error reports provider, I can imagine to disable the error reporter for this package in the beginning. Disabling would mean that the user would have to go to the preferences page and enable it explicitly by changing a value in some combo box.


Three options I’d feel comfortable with:

1. The milquetoast option: Disable error reporting for JEE in the first days (via preferences as outlined above). If no outages occur, we can rebuild the JEE package and reenable it. We could also do the same for the Java package. With that we could slowly phase-in and see whether we can handle the traffic.

2. The daredevil option: Deactivate it in new EPP downloads as soon as we see that there is too much traffic. But then the sending clients are already out there...

3. I forgot option three. But I had one before I went to breakfast… Ah, here it is: The BOFH option: Use the firewall to block every, say, second access to the HTTP HEAD requests to the remote problems database. With that, create a controlled phasing in by selectively disabling half of the clients (randomly). Instead of blocking every second request, we may disable it for some hours of the days. Since this will disable the error reporter until restart we may temporarily disable the clients completely on the server-side w/o changing any EPP package.

What do you think?


> It appears to me that you didn't just write code; you've engineered a solution.

Credits for this go to all people involved in various discussions over the past milestones and their feature requests. See bug 457115 [1] as one great example - but there are many more.



Denis, BTW:
can we get more bandwidth for dev.eclipse.org/recommenders? :-)


Marcel


[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=457115

> On 12 Jun 2015, at 20:26, Denis Roy <denis.roy@xxxxxxxxxxx> wrote:
>
> It appears to me that you didn't just write code; you've engineered a solution.
>
> No need for crossed fingers.  I think we're in good shape.  Beer will be on me at EclipseCon.
>
> Have a nice weekend.
>
> Denis
>
>
>
> On 12/06/15 02:21 PM, Marcel Bruch wrote:
>> Denis,
>>
>> there are four (4) + one (1) mechanisms in place that will / can limit the traffic to dev.eclipse.org/recommenders:
>>
>> Level 1: A local, persistent, per-workspace history that remembers what was send before. If an error is logged that has the same fingerprint as an already sent report, it is skipped. This should prevent clients to send the same error over and over again.
>>
>> Level 2: A “known errors database” that get’s downloaded from eclipse.org on IDE startup. If an error is logged in the IDE which was marked as “ignored” in the known-errors database, it won’t be send. No network traffic will happen. The server-side generates this database on demand and set’s every problem to ignore that was reported by more than 50 users.
>>
>> Level 3: The “emergency” power OFF switch [1]. On startup, we check whether a certain system status URL is reachable. If that       fails, the system deactivates the reporter until next restart of Eclipse.
>>
>> Level 4: If Eclipse is started with ‑Dorg.eclipse.epp.logging.aeri.ui.skipReports=true, no errors will be sent. EPP packages may add this into the eclipse.ini in worst case.
>>
>> Level 5: I’m not sure what exactly would happen if the firewall blocks access the service. But I assume we catch every exception and log it gracefully as a warning. So this looks like Level 5 to me.
>>
>> However, the traffic may still be large - especially in the first days. In that case I’m happy to pull the plug via Option 3 or Level 4 by rebuilding the EPP packages. Or 5 if you don’t see any other option. Or we start with just one EPP (e.g. Committers and or Java) package for the release and add more until we are sure it works. I think we have all options in your hands to control it on various levels.
>>
>>
>> Regarding Reporting to Bugzilla: Yes, the traffic goes to dev.eclipse.org/recommenders/* If this server is shut down, nothing can ever go to Bugzilla.
>>
>> I hope I'll keep my account for a while… still crossing fingers, though.
>>
>> Best,
>> Marcel
>>
>>
>> [1] https://git.eclipse.org/c/epp/org.eclipse.epp.logging.git/tree/bundles/org.eclipse.epp.logging.aeri.ui/src/org/eclipse/epp/internal/logging/aeri/ui/log/CheckServerAvailabilityJob.java#n51
>>
>>
>>> On 12 Jun 2015, at 19:57, Denis Roy <denis.roy@xxxxxxxxxxx> wrote:
>>>
>>>
>>>>>
>>>>> On 06/12/2015 01:15 AM, Marcel Bruch wrote:
>>>>>> We are now looking forward to the first two weeks of the Mars release
>>>>>> and cross fingers that we don’t get flooded.
>>>>>
>>>>> Marcel,
>>>>>
>>>>> Do we have an OFF switch somewhere so that we can turn these things off if we see we're going to get flooded?
>>>>
>>>> Have we...?
>>>> Do you...?
>>>
>>>
>>> I have a big "OFF" switch called a firewall, but I'm not sure how that will manifest itself in Eclipse if the error reporter cannot connect to its home site.
>>>
>>> Also, just so I understand, the 1000's of daily reports  (which could become 100,000's of daily reports on June 25) are all reports against the recommenders server, right?  Not Bugzilla bugs, right? The latter could see the OFF switch extended to your user account :)
>>>
>>> I will be at ECE to collect on any wrongdoings you do to your fellow committers.
>>>
>>> In all seriousness, let me know if there's anything I can do to make sure we're prepared for what is about to come.
>>>
>>>
>>> Denis
>
> _______________________________________________
> ide-dev mailing list
> ide-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/ide-dev

--
Codetrails GmbH
The knowledge transfer company

Robert-Bosch-Str. 7, 64293 Darmstadt
Phone: +49-6151-276-7092
Mobile: +49-179-131-7721
http://www.codetrails.com/

Managing Director: Dr. Marcel Bruch
Handelsregister: Darmstadt HRB 91940

_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/epp-dev