Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cdt-dev] Eclipse/CDT, GDB and MacOS 11.4/BigSur

Yeh, I’m peeling the layers of issues back.  
Yeh, that same program worked fine on 2020/06 Eclipse/CDT with GDB 9.2, its a very simple program.


Btw, do we(Eclipse/CDT) have a BuildBot like GDB?

Sam


Ps, There’s definitely a state of inertia if only due to logistics :) - just kidding around - after all, it’s what I signed up for :) :) :). I always knew I was stupid :) .:).  Just call me Don,  Don Quixote that is  :) :).


On Jun 29, 2021, at 8:15 AM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:

Hi Sam,

The trace shows that CDT inserted breakpoint at main (30-break-insert -t -f main), then did a run (32-exec-run --thread-group i1), but that GDB never stopped at the breakpoint. It looks to me like GDB 9.2 is completely broken (for MI at least), assuming that the program is a simple program that should have hit main quickly enough.

You can try to reproduce that outside of Eclipse too, but based on the conversations happening on gdb's mailing list[1] it looks like you are progressing with master branch which is a good idea.


Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Mon, 28 Jun 2021 at 15:49, Sam Warner <samuel.r.warner@xxxxxx> wrote:
Hi Jonah,
   
   I found a working 9.2 GDB in a Parallel’s guest I had setup for Eclipse/CDT 2020/06.  I moved it over to the 2021/06 environment, and had mixed results:

1.) Good News:  null-pointer error doesn’t happen,
2.) the original issue of the MI state-machines seeming to get "out of sync” between GDB and Eclipse is back, but worse, 100% now, rather than 80-90%.  I attached the MI log from eclipse, and some screen captures showing what it appears like for the user.


   I am going to go debug the 10.2 null-pointer issue (offered my help on rooting out for the GDB community), after which it appears I’ll have a GDB/MI - Eclipse/CDT/MI issue to resolve.

Sam



On Jun 26, 2021, at 12:30 PM, Sam Warner <samuel.r.warner@xxxxxx> wrote:

Hi Jonah,

 Done .✔️

  I am really curious if anyone knows the writer of https://www.slant.co/versus/1667/2472/~xcode_vs_eclipse-cdt ?
  He seems to be using Eclipse on a Mac, and not having this GDB issue.  Maybe there’s some issue on my installation.

   Also, maybe I’m the issue - anyone have this functioning? 
   I’ve seen many posts where folks are having issues with GDB, and it appears give up, or where they resolved by downgrading GDB.  I am looking the combination of for GDB/10.2, with Eclipse 2021.06 and OS-x 11.4.  Previously I had 2020/06 with GDB 9.1 on some variant of OS-x last-year working (kinda unreliable, which prompted my attempt to join the group to aid).  Just checking I am not configured wrong.


Sam

On Jun 26, 2021, at 11:32 AM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:

Hi Sam,

You can try sending to gdb@xxxxxxxxxxxxxx (subscribe at https://sourceware.org/mailman/listinfo/gdb/) to get some attention on the bug.

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Sat, 26 Jun 2021 at 14:27, Sam Warner <samuel.r.warner@xxxxxx> wrote:
HI Jonah,
   Do you know folks in the GDB dev community? 

   Seems the bug is sitting in the unconfirmed state.  On a side, someone emailed me content on how to setup code-signing for GDB.  This prompted me to add that the binary is properly signed (install on OS-X requires this weird need).  Short of that, I guess my next step is to dive into the debugger source code to identify the issue.

Sam

On Jun 24, 2021, at 10:19 AM, Sam Warner <samuel.r.warner@xxxxxx> wrote:

https://sourceware.org/bugzilla/show_bug.cgi?id=28010

Sam

On Jun 24, 2021, at 8:52 AM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:

Hi Sam,

That is great, with that info it is time to raise that as a bug with Gdb folk. If they provide a workaround we can try to code that in on CDT side.

Jonah 

On Thu., Jun. 24, 2021, 10:51 Sam Warner, <samuel.r.warner@xxxxxx> wrote:
… more… really odd part is if one ignores it, things function.

(gdb) file /Users/sam/eclipse-workspace/Eclipse.HelloWorld.MacOS.Gcc/Debug/Eclipse.HelloWorld.MacOS.Gcc
Reading symbols from /Users/sam/eclipse-workspace/Eclipse.HelloWorld.MacOS.Gcc/Debug/Eclipse.HelloWorld.MacOS.Gcc...
../../gdb/thread.c:95: internal-error: struct thread_info *inferior_thread(): Assertion `current_thread_ != nullptr' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) n

This is a bug, please report it.  For instructions, see:

../../gdb/thread.c:95: internal-error: struct thread_info *inferior_thread(): Assertion `current_thread_ != nullptr' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) n
Command aborted.
(gdb) b main
Breakpoint 1 at 0x10000270f: file ../src/Eclipse.HelloWorld.MacOS.Gcc.temp.cpp, line 22.
(gdb) bl
Undefined command: "bl".  Try "help".
(gdb) run
Starting program: /Users/sam/eclipse-workspace/Eclipse.HelloWorld.MacOS.Gcc/Debug/Eclipse.HelloWorld.MacOS.Gcc 
[New Thread 0x2203 of process 4658]
[New Thread 0x1b03 of process 4658]
warning: unhandled dyld version (17)

Thread 2 hit Breakpoint 1, main (argc=1, argv=0x7ffeefbffa68) at ../src/Eclipse.HelloWorld.MacOS.Gcc.temp.cpp:22
22 mainPhilosopher();


On Jun 24, 2021, at 7:46 AM, Sam Warner <samuel.r.warner@xxxxxx> wrote:

Hi Jonah,

   Repro’ed without using Eclipse/CDT:

   sam@eclipsec plugins % gdb
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin20.3.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) file --thread-group i1 /Users/sam/eclipse-workspace/Eclipse.HelloWorld.M\
acOS.Gcc/Release/Eclipse.HelloWorld.MacOS.Gcc
i1: No such file or directory.
(gdb) file /Users/sam/eclipse-workspace/Eclipse.HelloWorld.MacOS.Gcc/Debug/Eclipse.HelloWorld.MacOS.Gcc
Reading symbols from /Users/sam/eclipse-workspace/Eclipse.HelloWorld.MacOS.Gcc/Debug/Eclipse.HelloWorld.MacOS.Gcc...
../../gdb/thread.c:95: internal-error: struct thread_info *inferior_thread(): Assertion `current_thread_ != nullptr' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) 


On Jun 23, 2021, at 9:37 PM, Sam Warner <samuel.r.warner@xxxxxx> wrote:

Hi Jonah,
   Wow, I now feel less stupid :)..   Debug Perspectives is the way to turn on “GDB Trace” perspective, and then in the console window (after selecting the gdb trace console ) the following shows up(below).

Sam

08,896 2-list-thread-groups
208,909 3-gdb-version
208,914 2^done,groups=[{id="i1",type="process"}]
208,929 (gdb) 
208,939 ~"GNU gdb (GDB) 10.2\n"
208,941 ~"Copyright (C) 2021 Free Software Foundation, Inc.\n"
208,941 ~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is fre\
e software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitt\
ed by law."
208,943 ~"\nType \"show copying\" and \"show warranty\" for details.\n"
208,943 ~"This GDB was configured as \"x86_64-apple-darwin20.3.0\".\n"
208,944 ~"Type \"show configuration\" for configuration details.\n"
208,944 ~"For bug reporting instructions, please see:\n"
208,945 ~"Find the GDB manual and other documentation resources online at:\n    <http://www.gnu.org/\
software/gdb/documentation/>."
208,946 ~"\n\n"
208,947 ~"For help, type \"help\".\n"
208,947 ~"Type \"apropos word\" to search for commands related to \"word\".\n"
208,947 3^done
208,963 (gdb) 
208,972 4-environment-cd /Users/sam/eclipse-workspace/Eclipse.HelloWorld.MacOS.Gcc
208,985 4^done
208,986 (gdb) 
208,990 5-gdb-set breakpoint pending on
208,995 5^done
208,995 (gdb) 
209,003 6-gdb-set detach-on-fork on
209,010 6^done
209,010 (gdb) 
209,024 7-enable-pretty-printing
209,027 7^done
209,028 (gdb) 
209,033 8-gdb-set python print-stack none
209,037 8^done
209,037 (gdb) 
209,038 9-gdb-set print object on
209,043 9^done
209,043 (gdb) 
209,047 10-gdb-set print sevenbit-strings on
209,052 10^done
209,052 (gdb) 
209,055 11-gdb-set host-charset UTF-8
209,063 11^done
209,063 (gdb) 
209,070 12-gdb-set target-charset UTF-8
209,073 12^done
209,074 (gdb) 
209,076 13-gdb-set target-wide-charset UTF-32
209,086 13^done
209,086 (gdb) 
209,090 14-gdb-set dprintf-style call
209,094 14^done
209,095 (gdb) 
209,105 15source /Users/sam/eclipse-workspace/.gdbinit
209,110 &"source /Users/sam/eclipse-workspace/.gdbinit\n"
209,116 =cmd-param-changed,param="startup-with-shell",value="off"
209,117 15^done
209,117 (gdb) 
209,119 16-gdb-set target-async off
209,124 16^done
209,124 (gdb) 
209,129 17-gdb-set record full stop-at-limit off
209,136 17^done
209,136 (gdb) 
209,139 18-gdb-set auto-solib-add on
209,145 18^done
209,146 (gdb) 
209,236 19unset env 
209,237 20-gdb-set env inferior = --init-eval-command \"set startup-with-shell off\"
209,237 21-gdb-set env quiet = -q
209,239 &"unset env \n"
209,239 ~"Delete all environment variables? (y or n) [answered Y; input not from terminal]\n"
209,239 19^done
209,239 (gdb) 
209,246 20^done
209,246 (gdb) 
209,246 21^done
209,246 (gdb) 
209,248 22-file-exec-and-symbols --thread-group i1 /Users/sam/eclipse-workspace/Eclipse.HelloWorld.M\
acOS.Gcc/Release/Eclipse.HelloWorld.MacOS.Gcc
209,260 ~"../../gdb/thread.c:95: internal-error: struct thread_info *inferior_thread(): Assertion `c\
urrent_thread_ != nullptr' failed.\nA problem internal to GDB has been detected,\nfurther debugging \
may prove unreliable."
209,267 ~"\nQuit this debugging session? "
209,267 ~"(y or n) [answered Y; input not from terminal]\n"
209,268 &"\nThis is a bug, please report it."
209,268 &"  For instructions, see:\n"
209,268 ~"../../gdb/thread.c:95: internal-error: struct thread_info *inferior_thread(): Assertion `c\
urrent_thread_ != nullptr' failed.\nA problem internal to GDB has been detected,\nfurther debugging \
may prove unreliable."
209,273 ~"\nCreate a core file of GDB? "
209,280 ~"(y or n) [answered Y; input not from terminal]\n"



Sam

On Jun 23, 2021, at 7:43 PM, Sam Warner <samuel.r.warner@xxxxxx> wrote:

Hi Jonah,

   Sadly - there Eclipse/CDT ability to display the MI commands seems to be affected by the inability to start the debugger properly.  Below are some of the screen captures of that I am seeing.  Do you know the instructions to use Wireshark to monitor the MI traffic?

<Screen Shot 2021-06-23 at 7.39.13 PM.png><Screen Shot 2021-06-23 at 7.39.07 PM.png>
<Screen Shot 2021-06-23 at 7.38.46 PM.png>
   

 
Sam
Ps, I’m trying to get a MacPorts distribution of 9.2 (awaiting instructions ) if possible or rebuild if not.  



On Jun 23, 2021, at 5:11 PM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:

Hi Sam, 

The MI trace will be pretty short, 10 to 20 commands is my guess.

Jonah 

On Wed., Jun. 23, 2021, 19:26 Sam Warner, <samuel.r.warner@xxxxxx> wrote:
Hi Jonah,
  Yep - no action - just status of my analysis of the issue, though it’s nice to see that’s translated to a change in the CDT startup options for Eclipse/CDT on Mac.

  Yep - I’ll be taking traces of the MI flows - and pass on after I collect, as well as analyze in parallel.  Does this “matters” spot have a storage locations to put the data (as I am imagining the trace will be large).

Sam


On Jun 23, 2021, at 8:08 AM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:

Hi Sam,

Sounds like you are making good progress. I am not sure if there was an open question you had in there, if so, please let me know.

Here are some general comments:
- Using an external tools configuration is equivalent to running GDB outside of Eclipse as it doesn't involve CDT at all in that case.
- The "set startup-with-shell off" option can/should be added to CDT's startup sequence, it can be made optional by adding it as an option if needed, just like I did for "set new-console on" on Windows (see Bug 520580 and https://wiki.eclipse.org/CDT/User/NewIn94#Debug)
- I don't expect any difference between mi2 vs mi3 for this issue as the code for mi2 and mi3 in GDB is the same, other than breakpoint output handling. Once you have a look at the MI logs it should be easier to know for sure.

Jonah


~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Wed, 23 Jun 2021 at 01:57, Sam Warner <samuel.r.warner@xxxxxx> wrote:
Hi,
  - Good news :), kinda :( … one can have Eclipse/CDT start and stop GDB 10.2 on MacOS, and start inferior threads (processes attached to by gdb).  Trouble is, the entire GDB context is decoupled from Eclipse/CDT, as the MI interface isn’t used with the technique.   Just to see Eclipse/CDT - GDB working, I did the following:

1) created an external tools configuration, and used the variables section to manually give the name of the application of the Eclipse/CDT project
<Screen Shot 2021-06-22 at 10.26.31 PM.png>

2) ran the gdb and was able to control the application with break-points and execution control commands.  The application was specifically loaded as an inferior thread, akin to what I think the MI technique is attempting. 
<Screen Shot 2021-06-22 at 10.26.07 PM.png>

   Problems encountered needing resolution for the MI managed Local/Attach scenarios of Eclipse/CDT … with the results from these issues that now Eclipse/CDT on MacOS/BigSur with GDB (even LLDB)  seems completely unusable as an IDE.  One has to fall back to a somewhat decoupled development/debug process if they are to use these tools. 

a. Startup commands of -nx stop the loading of user scripts, yet - appears users scripts have been prescribed as a way to deal with inferior threads starting from the GDB startup sequence Eclipse/CDT has been using in the past.
<Screen Shot 2021-06-22 at 7.24.52 PM.png>
b. Gdb instructions for code-signing appear to need the ability to have instructions changed to enable debugging ‘attached’ processes
<Screen Shot 2021-06-22 at 10.20.34 PM.png>

c. Using either local or attach-process Debug configurations results in GDB null-ptr issue
<Screen Shot 2021-06-22 at 4.44.56 PM.png>

d. Attaching to a process with Eclipse/CDT results in the following (likely due to item ‘b’ above, yet message is not clearly stating
<Screen Shot 2021-06-22 at 10.16.29 PM.png>

e. Eclipse/CDT has no way to add command lines options to the GDB, and ’nx’ disables scripts.  Accidentally realized this when I was seeing the GDB behavior with command lines, and realizing the Debug Console wasn’t showing the same.  Finally realizing that the options I added after the ‘gdb’ executable are not put in the command line that starts the GDB process.

f. Some MI2 issue, that I still need to obtain logs to diagnose the sequence that leads to the null-ptr in GDB


Sam


— prior — 

On Jun 22, 2021, at 8:29 PM, Sam Warner via cdt-dev <cdt-dev@xxxxxxxxxxx> wrote:

Hi Jonah,

  Well  "-ix /Users/sam/eclipse-workspace/gdbinit” added to the command line enables the “set startup-with-shell off” to be given despite the “—nx”.  Doesn’t work around the GDB null-ptr issue sadly.  Something else from Eclipse/CDT startup is forcing the inferior thread issue of GDB.

  Worstcase, soon I’ll have a build environment for Eclipse/CDT and can try the Mi3, along with analyze the MI communications from the log.

Sam


On Jun 22, 2021, at 7:31 PM, Sam Warner via cdt-dev <cdt-dev@xxxxxxxxxxx> wrote:

Hi Jonah,

  While rebuilding I saw this little note (attached)
<Screen Shot 2021-06-22 at 7.24.52 PM.png>

  Roughly, the “NX” is stopping the “.gdbinit” override, and forcing the starting thread to be inferior, which in turn is exposing the GDB issue (Assertion `current_thread_ != nullptr’ failed).  I am going to look for a command line to add to the startup to get the same effect as “set startup-with-shell off

Sam

Ps, the Mi override will be helpful for me if I repeat my prior issue of Eclipse/CDT and GDB’s state-machines getting out-of-sync.  The logging info you provided will too. I’m gambling that Mi3 is backward compatible, and that the above bug will be easier to resolve if we can reproduce in Mi3




On Jun 22, 2021, at 6:59 PM, Sam Warner via cdt-dev <cdt-dev@xxxxxxxxxxx> wrote:

Hi Jonah,

  - great info - thank you.

  Yep, I confirmed the issue is only presently happening when Eclipse/CDT starts GDB.  It’s a GDB issue though.  I’ll use the traces to look into this further.  Someone’s posted a patch for this issue on GDB, and yep, I will need to a GDB image to get this, as I am pulling from Homebrew.

   Any way - manually - for me to configure to use ‘mi3’ when Eclipse/CDT launches GDB just to see?

Sam
Ps, I’ll follow the other information shortly (matters, setup build env, filing bug (if after building with the patch things aren’t resolved).
Pps, I’ll try lldb too



On Jun 22, 2021, at 6:35 PM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:

Hi Sam,

Not a problem at all. Let me see if I can answer your queries inline below.

On Tue, 22 Jun 2021 at 20:03, Sam Warner <samuel.r.warner@xxxxxx> wrote:
Hi Jonah,
 
   Apologize for missing this months meeting.  I’ve made some progress, though definitely this has been a bit of a challenge - despite getting to what seems like a productive point.  Seems Eclipse/CDT’s is starting up GDB with “—interpreter mi2 and —nx”.  

That is as expected :-)
 

  The terminated process shows this info on the MacOS (haven’t tried other OSes).  The net of the ‘mi2’ is it appears to not couple to GDB v10.2 properly, I keep getting:

“Gdb/thread.c:95: internal-error: struct….” 

  as an error message when I start a debug session with a “Local” app.  While I am certain we have exposed some GDB flaw, at the moment I am trying to scope out the issue a bit. (See attachments below).

Yes, nothing we have done in CDT should cause GDB to have an internal error, and if possible the bug should indeed be reported to GDB or, if the package was built by someone else, to them (e.g. if you got it from homebrew and the have their own patches that may need to be first port of call - https://formulae.brew.sh/formula/gdb)
 

   Any way to alter this to other MI interfaces manually, to see if another just works - before I dive deeper?

Background: MI is short for machine interface (just in case you didn't know). The 2 is the version number of the protocol. GDB has been on version 2 for a long time, but recently added a version 3. Version 3 is mostly the same as version 2, but with some protocol errors from version 2 fixed. See https://sourceware.org/gdb/current/onlinedocs/gdb/GDB_002fMI-Development-and-Front-Ends.html 

CDT can only talk to GDB with version mi2. What I do when trying to track down such problems is use the GDB CLI (start gdb without --interpreter argument) to reproduce similar sequences to make sure things work properly.

To create a bug report for GDB you may want to create the minimal sequence of commands that exposes the problem. If you want to see what CDT is sending/receiving with GDB you can turn on the MI traces view (see https://wiki.eclipse.org/CDT/User/NewIn92#Hide_gdb_traces_by_default) then you can start gdb in a terminal (with  --interpreter mi2)




Sam

Ps, does the team have Slack/Teams or some other equivalent method for questions/answers ?

There is a very underused (by CDT) mattermost channel https://mattermost.eclipse.org/eclipse/channels/cdt-developers - we can certainly chat on there.
 

pps, I am going to need some info on the build setup/environment for Eclipse/CDT too, so any pointers?

https://wiki.eclipse.org/Getting_started_with_CDT_development - Try the Oomph instructions first (https://wiki.eclipse.org/Getting_started_with_CDT_development#Using_Oomph) if you are not familiar with setting up Eclipse dev environments.
 

ppps, The “—nx” is rather interesting, as I spent quite a while trying to follow the exact directions for installing and coupling GDB to Eclipse/CDT, and the startup command disables the entire startup-script.  The interesting part being I kept trying to debut why the startup-script wasnt being used… but … hey…. Call me stupid :).  I could have just asked.

The --nx is to ensure nothing from the "external" environment affects the GDB/CDT interface. Many normal settings in .gdbinit files can cause problems. Individual launches can have init files (specified in the launch configuration) and that may be a good place to add the extra items like "set startup-with-shell off". 
 

<Screen Shot 2021-06-22 at 4.44.56 PM.png>

 What will be interesting about the above screenshot is what sequence of MI commands leads to that assertion.
 
<Screen Shot 2021-06-22 at 4.39.13 PM.png>

Cool to see new features of CDT being used already - https://wiki.eclipse.org/CDT/User/NewIn103#Debug

Let me know what else I can do to help you.

Jonah


 


On Apr 9, 2021, at 10:26 AM, Sam Warner via cdt-dev <cdt-dev@xxxxxxxxxxx> wrote:

Hi Johan,
   I’d be interested in helping, sorry I hadn’t helped before on the GDB issues I had identified with OS-X.  I’ll join the meeting to see how I might be of help, or if you know way beforehand - please do reply back.

Thanks,
Sam


On Apr 9, 2021, at 7:27 AM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:

Hello folks,

See you all on Wednesday for our monthly call:

CDT Monthly Call April 14 2021

  1. Welcome and sign yourself in
  2. Actions from last meeting
  3. CDT 10.3 / 2020-06 planning (M1 is on 12 April)
  4. Increasing minimum version of GDB we test and support “officially” Bug 572582
  5. Reenabling long disabled DSF-GDB tests (these were initially disabled when we moved to Jiro)
  6. Status of cdt-gdb-adapter
  7. Any other business?


~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev












Back to the top