Bug 526437 - Android SWT Libraries need to be forked
Summary: Android SWT Libraries need to be forked
Status: NEW
Alias: None
Product: andmore
Classification: Tools
Component: General (show other bugs)
Version: 0.5.1   Edit
Hardware: PC Windows 7
: P3 major (vote)
Target Milestone: 0.5.2   Edit
Assignee: Andrew Bowley CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-10-24 19:02 EDT by Andrew Bowley CLA
Modified: 2017-12-02 03:19 EST (History)
1 user (show)

See Also:


Attachments
Files required to compile sdkuilib with sdklib version 25 (9.15 KB, text/plain)
2017-10-24 19:02 EDT, Andrew Bowley CLA
no flags Details
New sdktool screenshot (259.54 KB, image/jpeg)
2017-11-14 05:06 EST, Andrew Bowley CLA
no flags Details
Old sdkmanager screen shot (203.79 KB, image/jpeg)
2017-11-14 05:07 EST, Andrew Bowley CLA
no flags Details
SDK Manager displays package types: extras and generic (189.34 KB, image/jpeg)
2017-11-28 05:26 EST, Andrew Bowley CLA
no flags Details
Package type selection dialog (28.89 KB, image/jpeg)
2017-11-28 05:27 EST, Andrew Bowley CLA
no flags Details
Individual package installer dialog (61.45 KB, image/jpeg)
2017-12-01 20:27 EST, Andrew Bowley CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Bowley CLA 2017-10-24 19:02:43 EDT
Created attachment 271164 [details]
Files required to compile sdkuilib with sdklib version 25

The Android SWT libraries on the AOSP site have become obsolete and incompatible with Android SDK Tools version 25. The most significant issue is that a lot of code has been removed from the sdklib library and the sdkuilib module can only be built by putting this code back in (156 source files). Once this is done, API changes prevent the code working at runtime, even though it compiles cleanly. A fork allows appropriate changes to be undertaken to restore essential Andmore functionality.
Comment 1 David Carver CLA 2017-10-27 12:50:07 EDT
Great.  Can you open a pull request, and include the new bundle in the main Andmore build.   This way we it is built when the Andmore build is run.  

Open a Pull Request, and we can start getting this integrated and the dependencies submitted to eclipse IP for review.
Comment 2 Andrew Bowley CLA 2017-10-28 23:59:06 EDT
I have submitted pull request https://github.com/xhteam/tools-motodev/pull/3
I'm not sure if I should have changed the default LHS of the pull request, but I am sure it's not hard to correct if that was a mistake.

The Andmore Maven build completes with the new SWT addition, but not surprizing, with integration test errors. There are errors in the original Andmore modules when imported into Eclipse and I have not tried to fix this as the focus is on the SWT component.
Comment 3 David Carver CLA 2017-10-31 08:40:14 EDT
(In reply to Andrew Bowley from comment #2)
> I have submitted pull request https://github.com/xhteam/tools-motodev/pull/3
> I'm not sure if I should have changed the default LHS of the pull request,
> but I am sure it's not hard to correct if that was a mistake.
> 
> The Andmore Maven build completes with the new SWT addition, but not
> surprizing, with integration test errors. There are errors in the original
> Andmore modules when imported into Eclipse and I have not tried to fix this
> as the focus is on the SWT component.

Andrew... It looks like you submitted the pull request to the wrong project.  Make sure to submit it to.

https://github.com/eclipse/andmore

Once that is done I'll take a look at the PR.
Comment 4 Eclipse Genie CLA 2017-11-01 17:20:51 EDT
GitHub Pull Request 121 created by [Kys3rK1ng]
https://github.com/eclipse/andmore/pull/121
Comment 5 Andrew Bowley CLA 2017-11-05 15:19:13 EST
This fork requires a second stage to remove legacy SDK code from the sdkuilib module. Until this is done, Andmore users will not be able to view and edit Android devices or SDK packages. This also impacts the ability to launch applications under development as a device selection dialog fails, resulting in an error message suggesting the user select a compatible device.

I have investigated the task and am proceeding by recreating the sdkuilib module in a piecemeal fashion. I found that to work on the module as a whole is hard as errors cascade once editing the code is under way. The first milestone is to get the sdkmanager application running with the rebuilt sdkuilib module. Once this is done I hope people more expert in Android than I am can review the screens and suggest what more can be done to bring them up to date as well as looking for bugs.
Comment 6 David Carver CLA 2017-11-13 08:52:32 EST
Andrew, I'll starting taking a look at this today.  And will try and get the CQ submitted so we can get the parallel review started on this.
Comment 7 Andrew Bowley CLA 2017-11-14 05:06:08 EST
Created attachment 271453 [details]
New sdktool screenshot
Comment 8 Andrew Bowley CLA 2017-11-14 05:07:01 EST
Created attachment 271454 [details]
Old sdkmanager screen shot
Comment 9 Andrew Bowley CLA 2017-11-14 05:07:59 EST
I have reached a milestone today by installing the NDK package in my local Android SDK using the replacement sdkmanager, which I have imaginatively named "sdktool". I have attached screenshots of my local packages as displayed by both sdktool and sdkmanager. The latter is broken in that it will not display packages with updates without also displaying all new packages, so I just show installed packages. It should be immediately obvious that the package names in the new application are more descriptive. 

I am nearing completion of the "first cut" and the main thing outstanding is to add AVD Manager operations Create, Start, Edit Repair, Delete and Details. Dealing with devices is much more staight forward than dealing with packages.

The sdktool is like sdkmanager in that it has a small Main class, with the rest consisting of a skduilib module and a large number of dependency jars. The updated sdkuilib module currently has 70 java files compared to the pull request's 195 files. There are also 24 xsd files containing 507K bytes which are due to be deleted.

I investigated what happened to Android Studio, but could only find a continuation of the deprecated classes, even in the latest Canary release of 3rd November. The only examples of using the new SDK libraries I have come across are in the libraries themselves, where there are two console applications, one for managing applications and one for devices. Without these examples to work from, I would not have been able to proceed.
Comment 10 David Carver CLA 2017-11-14 11:42:34 EST
(In reply to Andrew Bowley from comment #9)
> I have reached a milestone today by installing the NDK package in my local
> Android SDK using the replacement sdkmanager, which I have imaginatively
> named "sdktool". I have attached screenshots of my local packages as
> displayed by both sdktool and sdkmanager. The latter is broken in that it
> will not display packages with updates without also displaying all new
> packages, so I just show installed packages. It should be immediately
> obvious that the package names in the new application are more descriptive. 
> 
> I am nearing completion of the "first cut" and the main thing outstanding is
> to add AVD Manager operations Create, Start, Edit Repair, Delete and
> Details. Dealing with devices is much more staight forward than dealing with
> packages.
> 
> The sdktool is like sdkmanager in that it has a small Main class, with the
> rest consisting of a skduilib module and a large number of dependency jars.
> The updated sdkuilib module currently has 70 java files compared to the pull
> request's 195 files. There are also 24 xsd files containing 507K bytes which
> are due to be deleted.
> 
> I investigated what happened to Android Studio, but could only find a
> continuation of the deprecated classes, even in the latest Canary release of
> 3rd November. The only examples of using the new SDK libraries I have come
> across are in the libraries themselves, where there are two console
> applications, one for managing applications and one for devices. Without
> these examples to work from, I would not have been able to proceed.

Sounds like good progress, and welcome to the wonderful world of trying to figure out how to work with the undocumented Android SDK libraries.
Comment 11 Andrew Bowley CLA 2017-11-21 18:28:32 EST
The next milestone has been achieved, which is to be able to launch an application under development. I am using an "Android Application" run configuration targeting an actual device and configured to launch the Default Activity. To achieve this, I had to merge into the pull request the backlog of Andmore changes from Matthew Piggott and myself in order to integrate the new SWT bundles into Andmore. 

This step led to more intense development activity on both the SWT bundles and Andmore. The first main challenge was to move IDE windows out of Andmore "ADT" and into sdkuilib. The aim was to achieve a one-way dependency from sdklib to sdkuilib and avoid exposing the internal sdkuilib packages.
 
A second main challenge was to apply appropriate coding standards to the loading of images from the file system. The images were formerly being extracted from jars using a classloader and cached by the application. 
There is now a new image loader wich uses a JFace local resource manager and extracts images from bundles using a utility method provided by the Eclipse Platform. The new image loader is also now wired as a normal dependency instead of being accessed everywhere using a static class method.

What comes next is moving to a local build of Sequoyah to support ongoing development and then assessing the state of health of Andmore at the "leading edge". For starters, I want to look at a facility to download and install individual SDK packages which looks to be superfluous, even dangerous. 
I did not complete the revision of this facility pending further investigation. If it is required after all, then the package updater implementation will need a significant update to include package filtering.
Comment 12 Andrew Bowley CLA 2017-11-28 05:22:16 EST
The new SDK Manager now has package filtering as illustrated by the attached images. The packages are filtered on types introduced in the reworked SDK API. 
New packages are also now by default filtered so only the latest version is displayed. This allows for much greater clarity in presenting package details.

I finished translating the individual package install window to the new SDK API and used it to install a new SDK platform. This proved it is possible to install a new SDK from scratch using the new SDK API. 

However, the user experience was disappointing, with only a small window to look at containing only a progress bar and a status message line.  I suggest we instead employ the regular SDK Manager, using the package filter facility to provide the initial selection. Then the user can. before hitting the "Install" button, adjust the selection according to needs and experience. 

I have found and fixed a number of bugs in the SWT fork shake out and the changes can be committed at any time. However I have also encountered what I classify as "design issues", one going back to the era of Android API 8, which are clearly outside the scope of this bug report.
I will be creating a new bug report to track these and also hopefully get some input from other Andmoreans on the best way to proceed. The bug summary will be something like "Andmore design changes required for API 27".
Comment 13 Andrew Bowley CLA 2017-11-28 05:26:04 EST
Created attachment 271665 [details]
SDK Manager displays package types: extras and generic
Comment 14 Andrew Bowley CLA 2017-11-28 05:27:19 EST
Created attachment 271666 [details]
Package type selection dialog
Comment 15 David Carver CLA 2017-11-28 08:58:53 EST
This looks good.   Let me know if you need any help, or if we have anything that needs to go through the CQ process for IP approval.  i.e. any updated dependencies.   I'll try and remember today or tomorrow to get you added to the existing CQ.
Comment 16 Andrew Bowley CLA 2017-12-01 20:27:34 EST
Created attachment 271745 [details]
Individual package installer dialog
Comment 17 Andrew Bowley CLA 2017-12-02 03:19:16 EST
Comprehensively updated individual package installer dialog so it acts as a logger with cancel/OK button. See attached screen shot.
This dialog is not wired into Andmore yet, as it requires changes to the Andmore design which are out of scope for this bug report.

I have committed this change along with all the outstanding bug fixes etc and no more changes are anticipated until the pull request is completed. Note there 2 commits, the second fixing an issue with Git not retaining libs directories which become empty by removing the legacy SWT jars.

Following is a complete list of dependencies to assist with checking that nothing has been missed. I can provide more information, if required:

Andmore-swt Dependencies:

  Android libraries
    common-25.3.3.jar
    ddmlib-25.3.3.jar
    dvlib-25.3.3.jar
    layoutlib-api-25.3.3.jar
    repository-25.3.3.jar
    sdk-common-25.3.3.jar
    sdklib-25.3.3.jar

  Third party dependencies of Android libraries
    commons-compress-1.8.1.jar
    guava-18.0.jar
    httpmime-4.1.jar
    jimfs-1.1.jar
    kxml2-2.3.0.jar

  Third party dependencies of DDM - the **NEW** jars
    chart_swt-1.0.13.jar
    jcommon-1.0.24.jar
    jfreechart-1.0.19.jar
    jfreechart-swt-1.0.jar
    
Android-core Dependencies*:

  Android libraries:
    builder-2.3.3.jar
    manifest-merger-25.3.3.jar
    lint-25.3.3.jar
    lint-api-25.3.3.jar
    lint-checks-25.3.3.jar
  
  Third party dependencies of Android libraries
    asm-5.0.4.jar
    asm-analysis-5.0.4.jar
    asm-tree-5.0.4.jar
    lombok-ast-0.2.3.jar
    uast-162.2228.14.jar

  Applications from Android source repository
    asset-studio.jar
    ninepatch.jar
    propertysheet.jar
    rule-api.jar

  Dependencies of gldebugger
    host-libprotobuf-java-2.3.0-lite.jar
    liblzf-1.0.jar  
  
  Mystery - just a small Ant utility library - probably not needed
    ant-glob.jar

Third party bundles - no license issues assumed:

  org.freemarker;bundle-version="2.3.22",
  com.google.gson;bundle-version="2.2.4",
  org.apache.httpcomponents.httpclient;bundle-version="4.1.3",
  org.apache.httpcomponents.httpcore;bundle-version="4.1.4",
  org.apache.commons.logging;bundle-version="1.1.1",

*  The asm archives replaced bundle org.objectweb.asm containing an older asm library version
   The uast-162.2228.14.jar archive is new to Android and is provided by Google as an external library and maybe needs CQ.
   "A subset of IntelliJ IDEA's AST APIs and implementation, focusing on the read-only aspects, intended for use by **Android Lint** when running outside of the IDE (typically from Gradle.). Based on IntelliJ IDEA tag idea/162.2228.14."
 
 Android-core Dependencies:
   common dependency
     androidjdbc.jar (sqlite JDBC)
   
   certmanager dependencies:
     bcpkix-jdk15on-147.jar
     bcprov-jdk15on-147.jar
     
   logger dependency:
     log4j-1.2.14.jar