Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [ptp-dev] New SFTP requirement for remote?
PROXY connections will still use ssh, but will launch the proxy whenever multiple channels or file operations are required. Once you set the default to PROXY, you shouldn't need to change the remote services type again.

Regards,
Greg

On Oct 11, 2017, at 3:15 PM, Rocky Dunlap - NOAA Affiliate <rocky.dunlap@xxxxxxxx> wrote:

So, after setting the default connection type to "PROXY", I should set up the two-hop connections as SSH connections?  Or PROXY connections?

Rocky

On Wed, Oct 11, 2017 at 1:01 PM, Greg Watson <g.watson@xxxxxxxxxxxx> wrote:
It should already be in the Eclipse for Parallel Application Developers download for Oxygen. Click on "Remote Development" in Preferences and choose PROXY from the "Default connection type". You can then just set up a normal ssh connection or two-hop connection using the "New connection properties" dialog.

I just found a bug that prevents it working with openjdk :-( so you need to used the Oracle java for now.

Regards
Greg


On Oct 10, 2017, at 4:47 PM, Rocky Dunlap - NOAA Affiliate <rocky.dunlap@xxxxxxxx> wrote:

Great - please let me know how to try it out.  Is there a prebuilt binary or p2 site that I can point to?

Rocky

On Tue, Oct 10, 2017 at 12:58 PM, Greg Watson <g.watson@xxxxxxxxxxxx> wrote:
Hi Rocky,

The java program runs on the final host. As long as there's an ssh connection to the remote end it should work.

Greg

On Oct 10, 2017, at 12:02 PM, Rocky Dunlap - NOAA Affiliate <rocky.dunlap@xxxxxxxx> wrote:

Greg,

Thanks for proposing this option.  Does the java program need to run on the intermediate or the final host?

Rocky

On Mon, Oct 9, 2017 at 7:24 PM, Greg Watson <g.watson@xxxxxxxxxxxx> wrote:
Rocky,

Sorry, too busy lately. There's a new proxy service that only requires a single ssh channel and no sftp channels. It does require java on the remote system, however, as it downloads a small java app that handles the multiplexing and file service at the remote end. Let me know if you'd like to give that a try. 

Regards,
Greg




On Sep 24, 2017, at 4:29 PM, Rocky Dunlap - NOAA Affiliate <rocky.dunlap@xxxxxxxx> wrote:

Greg,

Any word on this?  This is an outstanding issue for a number of folks who connect through an intermediate node with port forwarding disallowed.

For the particular system in question, NASA's Discover, there is a procedure here to set up a "master" ssh connection using "ssh -M".  This allows clients to share the master connection.


You would set up your ~/.ssh/config something like this:

Host dali.nccs.nasa.gov
    User <your user name>
    ControlPath ~/.ssh/master-%r%h:%p
    ControlMaster no
    ServerAliveInterval 900
    ProxyCommand ssh login.nccs.nasa.gov direct %h
    LogLevel Quiet
    Protocol 2
    ConnectTimeout 300

This does work fine for command line tools (e.g., git, rsync, etc.) but I'm not sure if this approach could be used with Eclipse.  The Eclipse connection would need to be set up to use an established master connection.  Who would know whether/how to set up Eclipse to be a client of a master SSH connection?

Rocky

On Wed, Sep 13, 2017 at 3:46 PM, Rocky Dunlap - NOAA Affiliate <rocky.dunlap@xxxxxxxx> wrote:
Greg,

I wanted to see if it we can resolve this issue we discussed a while ago about 2-hop connections. I think you were able to correctly identify that our situation is a 2-hop connection with no forwarding and a restricted shell (see below).  You had proposed modifications to the openMinimal() method that would allow us to establish a connection.

Do you recall if any work had been done on this?  I just tried in the latest Oxygen release and I still get a RemoteConnectionException trying to use the two connection ssh proxy approach in Eclipse.

Rocky

On Thu, Sep 8, 2016 at 11:47 AM, Greg Watson <g.watson@xxxxxxxxxxxx> wrote:
The openMinimal() method Roland added was to support proxy connections where there is a 2-hop process for accessing the target system. The connection to the intermediate machine is just a very basic ssh connection that is used to either set up port forwarding (if the intermediate host allows it) or run a command (usually another ssh command). Once a connection is established to the target system, this connection is opened “fully”, which would then (try to) start an sftp server channel, etc.

Here are the situations that I think need to be supported:

1-hop host:
a) ssh + sftp, >2 channels
b) ssh + sftp, 2 channels 
c) ssh, >1 channels
d) ssh, 1 channel

2-hop intermediate host (same target host options as for 1-hop):
e) forwarding
f) no forwarding, normal shell
g) no forwarding, restricted shell

I’m presuming your system is (g). I think the connection class will need to be changed to deal with this as it still assumes that commands like ‘printenv’ are allowed.

Greg

On Sep 1, 2016, at 9:29 PM, Rocky Dunlap - NOAA Affiliate <rocky.dunlap@xxxxxxxx> wrote:

Greg,

I just tried with Mars 4.5.0 and in also shows the SFTP error.  Apologies, I thought it had worked in Mars.  Then I tried the Luna SR1 and it is also showing the same issue!

Maybe what is happening here is that the system I am connecting to previously had sftp available and it has since been disabled, so now these connections are failing in Luna - Neon, although they previously worked.  (The system in question is NASA Discover.)

I looked back at my interactions with Roland.  It resulted in these changes:


Maybe that will shed light on what changed under the hood.

It must have been that the issue he resolved had more to do with the limited shell--i.e., you cannot run any commands to pull in environment information, but the connection is still established.

Basically, for some of these login nodes they are stripped down and you cannot assume much of anything.  However, it would be preferable if PTP made a "best effort" to establish a connection -- i.e., don't fail if sftp or env. commands are not available.  I think Roland had an openMinimal() in there which is maybe doing just this.

Rocky



On Thu, Sep 1, 2016 at 1:13 PM, Greg Watson <g.watson@xxxxxxxxxxxx> wrote:
Rocky,

I was able to take a look at this today. I’m puzzled however, as I don’t see any changes between Mars and Neon that would be causing this to fail. Can you try a Mars installation to verify that it is working?

Thanks,
Greg

On Aug 22, 2016, at 3:14 PM, Rocky Dunlap - NOAA Affiliate <rocky.dunlap@xxxxxxxx> wrote:

Greg,

I was able to test the change.  Interestingly, I am able to connect to a login node (this is NASA Discover), although I still get an exception:

Thread [pool-3-thread-1] (Suspended (exception JSchException))
ChannelExec(Channel).sendChannelOpen() line: 728
ChannelExec(Channel).connect(int) line: 151
ChannelExec(Channel).connect() line: 145
ExecCommand$1.call() line: 38
ExecCommand$1.call() line: 1
FutureTask<V>.run() line: 266 [local variables unavailable]
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142
ThreadPoolExecutor$Worker.run() line: 617
Thread.run() line: 745

This is happening when trying to run the command "printenv" on the connection.  I remember going through this with Roland.  The issue here is that these login nodes have a restricted shell so you cannot run arbitrary commands.  The way I have it set up is that the login connection is used only as a proxy.  Roland had it so that it would not fail even if unable to execute this initial set of commands.

Since the connection was actually established (despite the exception above) I tried to use it as a proxy for another connection.  However, this fails with the following:

Thread [Worker-21] (Suspended (exception JSchException))
owns: JSchConnectionProxyFactory$CommandProxy  (id=288)
ChannelExec(Channel).sendChannelOpen() line: 728
ChannelExec(Channel).connect(int) line: 151
ChannelExec(Channel).connect() line: 145
JSchProcessBuilder.start(int) line: 184
JSchProcessBuilder(AbstractRemoteProcessBuilder).start() line: 157
JSchConnectionProxyFactory$CommandProxy.connect(SocketFactory, String, int, int) line: 108
Session.connect(int) line: 231
JSchConnection.newSession(IProgressMonitor) line: 781
JSchConnection.open(IProgressMonitor, boolean) line: 831
JSchConnection.open(IProgressMonitor) line: 804
OpenConnectionHandler$1.run(IProgressMonitor) line: 51
Worker.run() line: 55

Rocky

On Mon, Aug 22, 2016 at 12:07 PM, Greg Watson <g.watson@xxxxxxxxxxxx> wrote:
Gerrit change set is here: https://git.eclipse.org/r/79393

Let me know if you have problems testing it.

Regards,
Greg

On Aug 12, 2016, at 2:49 PM, Rocky Dunlap - NOAA Affiliate <rocky.dunlap@xxxxxxxx> wrote:


On Fri, Aug 12, 2016 at 12:43 PM, Greg Watson <g.watson@xxxxxxxxxxxx> wrote:
Yes, I don’t think this should be a requirement for general remote ssh connections. Obviously there will be services (e.g. synchronized projects) that will not work in this case. Please open a bug and I’ll take a look.

Thanks
Greg

> On Aug 12, 2016, at 2:30 PM, Rocky Dunlap - NOAA Affiliate <rocky.dunlap@xxxxxxxx> wrote:
>
> We have a number of our customers on government machines without sftp support on login nodes.  Would it be possible to relax the sftp requirement back to how it worked in Mars?  It would be a big help for us.
>
> Thread [Worker-12] (Suspended (exception JSchException))
> ChannelSftp.start() line: 315
> ChannelSftp(Channel).connect(int) line: 152
> ChannelSftp(Channel).connect() line: 145
> JSchConnection.openSftpChannel(Session) line: 1023
> JSchConnection.checkConfiguration(Session, IProgressMonitor) line: 331
> JSchConnection.open(IProgressMonitor, boolean) line: 1008
> JSchConnection.open(IProgressMonitor) line: 970
> OpenConnectionHandler$1.run(IProgressMonitor) line: 51
> Worker.run() line: 55
>
> Thanks,
> Rocky
> _______________________________________________
> ptp-dev mailing list
> ptp-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/ptp-dev

_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev

_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev


_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev

_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev


_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev

_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev


_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev


_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev


_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev

_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev


_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev

_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev




_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev

_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ptp-dev

Back to the top